Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data
1 like780 views
The document discusses a privacy-preserving extreme learning machine algorithm based on secure multi-party computation for vertically partitioned data. It outlines the challenges and contributions of the approach, including the construction of a global classification model from distributed datasets, while keeping individual inputs private. The work emphasizes the efficient computation of functions across multiple parties' private data with minimal communication cost.
![Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Conclusions
Conclusions
Summary
ELM learning algorithm, a new method.
ELM outperforms traditional Single Layer Feed-forward Neural-networks and
Support Vector Machines for Big Data 1
In all fields that ELM is applied (i.e. medical records, business, government),
privacy is a major concern.
Conclusions
Privacy-preserving learning model is proposed in vertically partitioned data
In multi-party partitioning without sharing the data of each site to
the others.
Master party divides weight vector, and each party calculates the activation
function result with its data and weight vector.
1Cambria, Erik, et al. ”Extreme learning machines [trends & controversies].”
Intelligent Systems, IEEE 28.6 (2013): 30-59.
Ferhat ¨Ozg¨ur C¸atak ozgur.catak@tubitak.gov.tr T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin](https://image.slidesharecdn.com/csmining2015fozgurcatak-151115055607-lva1-app6892/85/Secure-Multi-Party-Computation-Based-Privacy-Preserving-Extreme-Learning-Machine-Algorithm-Over-Vertically-Distributed-Data-24-320.jpg)
Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data
- 1. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, Turkey Nov. 10 2015 The 2015 International Data Mining and Cybersecurity Workshop With ICONIP2015 Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 2. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Table of Contents 1 Introduction Secure Multiparty Computation Contributions 2 Preliminaries ELM Secure Multi Party Computation Secure Multi-Party Addition 3 Privacy-preserving ELM over vertically partitioned data Privacy-preserving ELM 4 Experiments Experimental setup Simulation Results 5 Conclusions Conclusions Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 3. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Secure Multiparty Computation Contributions Table of Contents 1 Introduction Secure Multiparty Computation Contributions 2 Preliminaries ELM Secure Multi Party Computation Secure Multi-Party Addition 3 Privacy-preserving ELM over vertically partitioned data Privacy-preserving ELM 4 Experiments Experimental setup Simulation Results 5 Conclusions Conclusions Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 4. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Secure Multiparty Computation Contributions Secure Multiparty Computation (SMC) Definition SMC : The goal of creating methods for parties to jointly compute a function over their inputs while keeping those inputs private. In an MPC, a given number of participants, p1, p2, · · · , pN each has private data, respectively d1, d2, · · · , dN Participants want to compute the value of a public function on that private data: F(d1, d2, ..., dN ) while keeping their own inputs secret. Challenges : To protect each participant’s private data, and intermediate results The computation/communication cost introduced to each participant shall be affordable Training data is arbitrarily partitioned Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 5. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Secure Multiparty Computation Contributions Contributions Main contributions of the work Privacy-preserving Extreme Learning Machine (ELM) training model Global ELM classification model from the distributed data sets in multiple parties Training data set is vertically partitioned among the parties The final distributed model is constructed at an independent party Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 6. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Table of Contents 1 Introduction Secure Multiparty Computation Contributions 2 Preliminaries ELM Secure Multi Party Computation Secure Multi-Party Addition 3 Privacy-preserving ELM over vertically partitioned data Privacy-preserving ELM 4 Experiments Experimental setup Simulation Results 5 Conclusions Conclusions Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 7. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Extreme Learning Machine (ELM) Learning Without Iterative Tuning All hidden node parameters can be randomly generated without the knowledge of the training data. That is, for any continuous target function f and any randomly generated sequence limL→∞ ||f (x) − fL(x)|| = limL→∞ ||f (x) − L i=1 βi G(ai , bi , x)|| = 0 holds with probability one if βi is chosen to minimize ||f (x) − fL(x)||, ∀i G.-B. Huang, et al., ”Universal approximation using incremental constructive feedforward networks with random hidden nodes,” IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006. G.-B. Huang, et al., ”Convex incremental extreme learning machine,” Neurocomputing, vol. 70, pp. 3056-3062, 2007. G.-B. Huang, et al., ”Enhanced random search based incremental extreme learning machine,” Neurocomputing, vol. 71, pp. 3460-3468, 2008. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 8. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Extreme Learning Machine (ELM) Figure: Generalized SLFN Mathematical Model For N arbitrary distinct samples (xi , ti ) ∈ Rn × Rm, SLFNs with L hidden nodes each with output function, G(ai , bi , x) are mathematically modeled as L i=1 βi G(ai , bi , xj ) = tj , j = 1, ..., N (1) (ai , bi ): hidden node parameters. βi : the weight vector connecting the ith hidden node and the output node. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 9. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Extreme Learning Machine (ELM) Mathematical Model L i=1 βi G(ai , bi , xj ) = tj , j = 1, ..., N is equivalent to Hβ = T where, h(x1) ... h(xN) = G(a1, b1, x1) · · · G(aL, bL, x1) ... ... ... G(a1, b1, xN) · · · G(aL, bL, xN) N×L (2) β = βT 1 ... βT L L×m ve T = tT 1 ... tT N N×m (3) H, is called the hidden layer output matrix of the neural network, the ith column of H is the output of the ith hidden node with respect to inputs , x1, ..., xn Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 10. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Extreme Learning Machine (ELM) Three-Step Learning Model Given a training set D = {(xi , ti )|xi ∈ Rn , ti ∈ Rm , i = 1, · · · , N}, hidden node output function G(a, b, x), and the number of hidden nodes L, 1 Assign randomly hidden node parameters (ai , bi ), i = 1, · · · , L. 2 Calculate the hidden layer output matrix H. 3 Calculate the output weight β = Ht T. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 11. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Secure Multi Party Computation Secure Multi-Party Computation The problem of n players to compute an agreed function f of their inputs In vertically partitioned data, each party holds different attributes of same data set. The partition strategy is shown in Figure 2. x1,1 · · · x1,t−1 · · · x1,t · · · x1,k x2,1 · · · x2,t−1 · · · x2,t · · · x2,k ... ... ... ... ... ... ... xm,1 · · · xm,t−1 · · · xm,t · · · xm,k P1 · · · Pk D = Figure: Vertically partitioned data set D. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 12. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Secure Multi-Party Addition I Secure Multi-Party Addition Assume k ≥ 3 parties, P0, · · · , Pk−1, with party Pi holding value vi Aim is to compute v = k−1 i=0 vi Sum of vi lies in F P0 randomly chooses a number R ∈ F P0 adds R to its local value v0, Sends v0 = R + v0mod|F| to P1 The remaining parties Pi , i = 1, · · · , k − 1 Pi receives V = R + i−1 j=0 vj mod|F| Pi sends R + j=1 ivj mod|F| = (vi + V )mod|F| Finally, P0, subtracts R from the final message to compute actual results. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 13. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Secure Multi-Party Addition II Party0 x0 , R ∈ F Party1 x1 Party2 x2 Partyk xk V = R + x0mod|F| V = R + i−1 j=0 xjmod|F| (xi + V )mod|F| Figure: Secure Multi-Party Addition. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 14. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions ELM Secure Multi Party Computation Secure Multi-Party Addition Secure Multi-Party Addition III Algorithm 1: Secure multi-party addition 1: procedure SMA(P) 2: P0 : R ← rand(F) P0 randomly chooses a number R 3: V ← R + x0 mod F 4: P0 sends V to node P1 5: for i = 1, · · · , k − 1 do 6: Pi receives V = R + i−1 j=0 xj mod F 7: Pi computes V = R + i j=1 xj mod F = ((xi + V ) mod F) 8: Pi sends V to node Pi+1 9: P0 : V ← (V − R) = (V − R mod F) Actual addition result 10: end procedure Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 15. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Privacy-preserving ELM Table of Contents 1 Introduction Secure Multiparty Computation Contributions 2 Preliminaries ELM Secure Multi Party Computation Secure Multi-Party Addition 3 Privacy-preserving ELM over vertically partitioned data Privacy-preserving ELM 4 Experiments Experimental setup Simulation Results 5 Conclusions Conclusions Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 16. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Privacy-preserving ELM Privacy-preserving ELM I 1 Master party creates weight matrix, W ∈ RL×N 2 Master party distributes partition W with same feature size for each parties. 3 Party P0 creates a random matrix, R = rand1,1(F) · · · rand1,L(F) ... ... ... randN,1(F) · · · randN,L(F) N×L 4 Party P0 creates perturbated output, V = R + x0 1 · w0 1 + b0 1 · · · x0 1 · w0 L + b0 L ... ... ... x0 N · w0 1 + b0 1 · · · x0 N · w0 L + b0 L 5 for i = 1, · · · , k − 1 Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 17. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Privacy-preserving ELM Privacy-preserving ELM II Pi computes V = V + xi 1 · wi 1 + bi 1 · · · xi 1 · wi L + bi L ... ... ... xi N · wi 1 + bi 1 · · · xi L · wi N + bi N Pi sends V to Pi+1 6 P0 subtracts random matrix, R, from the received matrix V. H = (V − R) mod F 7 Hidden layer node weight vector, β, is calculated. β = H† · T Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 18. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Privacy-preserving ELM Privacy-preserving ELM III P arty0 x0 , w, H0 ,R ∈ F, Party1 x1 , H1 , w Party2 x2 , H2 , w Partyk xk−1 , Hk−1 , w V0 = R + H0 mod|F| V1 = V0+H1 mod|F| V2 = V1 + H2 mod|F| Vk = Vk−1 + Hk mod|F| Hi = G(w1, b1, xi 1) · · · G(wL, bL, xi 1) . . . ... . . . G(w1, b1, xi N ) · · · G(wL, bL, xi N ) N×L R = rand1,1(F) · · · rand1,L(F) . . . ... . . . randN,1(F) · · · randN,L(F) N×L H = (V − R) mod F Figure: Overview of privacy-preserving ELM learning. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 19. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Experimental setup Simulation Results Table of Contents 1 Introduction Secure Multiparty Computation Contributions 2 Preliminaries ELM Secure Multi Party Computation Secure Multi-Party Addition 3 Privacy-preserving ELM over vertically partitioned data Privacy-preserving ELM 4 Experiments Experimental setup Simulation Results 5 Conclusions Conclusions Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 20. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Experimental setup Simulation Results Experimental setup I Our approach is applied to six different data sets to verify its model effectivity and efficiency. Table: Description of the testing data sets used in the experiments. Data set #Train #Classes #Attributes australian 690 2 14 colon-cancer 62 2 2,000 diabetes 768 2 8 duke breast cancer 44 2 7,129 heart 270 2 13 ionosphere 351 2 34 Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 21. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Experimental setup Simulation Results Simulation Results I The number of party size, k: from 3 to number of feature n, of the data set For instance, k = 3, and n = 14, then the first two party have 5 attributes, and last party has 4 attributes. The accuracy of secure multi-party computation based ELM is exactly same for the traditional ELM training algorithm. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 22. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Experimental setup Simulation Results Simulation Results II Figure shows results of our simulations. Time scale becomes its steady state position when number of parties k, moves closer to number of attributes, n. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10 −1 10 0 10 1 10 2 10 3 AttributeS ize(n) P artyS ize(k) Timescale australian colon−cancer diabetes duke heart ionosphere Figure: Vertically partitioned data set D. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 23. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Conclusions Table of Contents 1 Introduction Secure Multiparty Computation Contributions 2 Preliminaries ELM Secure Multi Party Computation Secure Multi-Party Addition 3 Privacy-preserving ELM over vertically partitioned data Privacy-preserving ELM 4 Experiments Experimental setup Simulation Results 5 Conclusions Conclusions Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 24. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Conclusions Conclusions Summary ELM learning algorithm, a new method. ELM outperforms traditional Single Layer Feed-forward Neural-networks and Support Vector Machines for Big Data 1 In all fields that ELM is applied (i.e. medical records, business, government), privacy is a major concern. Conclusions Privacy-preserving learning model is proposed in vertically partitioned data In multi-party partitioning without sharing the data of each site to the others. Master party divides weight vector, and each party calculates the activation function result with its data and weight vector. 1Cambria, Erik, et al. ”Extreme learning machines [trends & controversies].” Intelligent Systems, IEEE 28.6 (2013): 30-59. Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin
- 25. Introduction Preliminaries Privacy-preserving ELM over vertically partitioned data Experiments Conclusions Conclusions Thank You Ferhat ¨Ozg¨ur C¸atak [email protected] Ferhat ¨Ozg¨ur C¸atak [email protected] T¨UB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin