![Labeled Directed Relationship
Gives
A B
(A) – [:Gives] - >
(B)](https://image.slidesharecdn.com/neo4jgraphconnectnyc-121002192227-phpapp02/85/Analyzing-FEC-Data-with-NEO4J-53-320.jpg)
![Variable Length Path
A B
A B
A B
(A) – [*] - > (B)](https://image.slidesharecdn.com/neo4jgraphconnectnyc-121002192227-phpapp02/85/Analyzing-FEC-Data-with-NEO4J-54-320.jpg)
![Simple Contribution
• start a = node(5058)
• match p = a-[:RECEIVES]-(b)<-
[:GAVE]-(c)
• where 1=1
• and b.contribAmt > 1
• return c
36](https://image.slidesharecdn.com/neo4jgraphconnectnyc-121002192227-phpapp02/85/Analyzing-FEC-Data-with-NEO4J-58-320.jpg)
![Bachman Contributions
// Bachman Contributions
start a = node(3946)
match p = a-[:RECEIVES]-(b)
where 1=1
and b.contribAmt > 1000
return b](https://image.slidesharecdn.com/neo4jgraphconnectnyc-121002192227-phpapp02/85/Analyzing-FEC-Data-with-NEO4J-59-320.jpg)
![Bachman Contributions
• start a = node(3946)
• match p = a-[:RECEIVES]-(b)<-
[:GAVE]-(c)
• where 1=1
• and (c.INDIVSTATE? = "MN")
• and b.contribAmt > 1
• return sum(b.contribAmt)
39](https://image.slidesharecdn.com/neo4jgraphconnectnyc-121002192227-phpapp02/85/Analyzing-FEC-Data-with-NEO4J-61-320.jpg)
![Julie Waddle
START person = node(191468)
MATCH person-[*]->c<-[*]-d
return c.contribAmt,
c.contribDate, d.name
40](https://image.slidesharecdn.com/neo4jgraphconnectnyc-121002192227-phpapp02/85/Analyzing-FEC-Data-with-NEO4J-62-320.jpg)
![SuperPac Examples
START n=node:committees('commName:
1911*')
MATCH n-[r:SUPERPACEXPEND]->c
RETURN c.PURPOSE, count(c.PURPOSE)
ORDER BY count(c.PURPOSE) DESC
41](https://image.slidesharecdn.com/neo4jgraphconnectnyc-121002192227-phpapp02/85/Analyzing-FEC-Data-with-NEO4J-63-320.jpg)
![More SuperPac Examples
START n=node:committees('commName:*')
MATCH n-[r:SUPERPACEXPEND]->c
WHERE c.PURPOSE! <> "ZOO"
RETURN c.PURPOSE, count(c.PURPOSE)
ORDER BY count(c.PURPOSE) DESC
START n=node:committees('commName:*')
MATCH n-[r:SUPERPACEXPEND]->c
WHERE c.PURPOSE! = "Rental Van"
RETURN c.PURPOSE, count(c.PURPOSE), c.CANDIDATE,
sum(c.expendAmt)
ORDER BY count(c.PURPOSE) DESC
42](https://image.slidesharecdn.com/neo4jgraphconnectnyc-121002192227-phpapp02/85/Analyzing-FEC-Data-with-NEO4J-64-320.jpg)
Analyzing FEC Data with NEO4J
- 1. Analyzing Federal Campaign Contributions Dave Fauth 1
- 2. About Me Full time consultant Playing with NoSQL and Neo4J for a couple of years • My Blog: http:// www.intelliwareness.org • Find me on Twitter: @davefauth • Email me: [email protected]
- 3. Agenda • Graph Database Overview • The Data • Data Layout • Data Prep • Analysis • Q&A
- 5. A Graph Database 5
- 6. A Graph Database ๏ no: not for charts & diagrams, or vector artwork 5
- 7. A Graph Database ๏ no: not for charts & diagrams, or vector artwork ๏ yes: for storing data that is structured as a graph 5
- 8. A Graph Database ๏ no: not for charts & diagrams, or vector artwork ๏ yes: for storing data that is structured as a graph • remember linked lists, trees? 5
- 9. A Graph Database ๏ no: not for charts & diagrams, or vector artwork ๏ yes: for storing data that is structured as a graph • remember linked lists, trees? • graphs are the general-purpose data structure 5
- 10. A Graph Database ๏ no: not for charts & diagrams, or vector artwork ๏ yes: for storing data that is structured as a graph • remember linked lists, trees? • graphs are the general-purpose data structure ๏ “A relational database may tell you the average age of everyone in this United States, but a graph database will tell you who is most likely to buy you a beer.” 5
- 11. A Graph Database 6
- 12. You know relational 7
- 13. You know relational 7
- 14. You know relational foo 7
- 15. You know relational foo bar 7
- 16. You know relational foo foo_bar bar 7
- 17. You know relational foo foo_bar bar 7
- 18. You know relational foo foo_bar bar 7
- 19. You know relational foo foo_bar bar 7
- 20. You know relational now consider relationships... 7
- 21. You know relational now consider relationships... 7
- 22. You know relational now consider relationships... 7
- 23. You know relational now consider relationships... 7
- 24. You know relational now consider relationships... 7
- 25. You know relational 7
- 26. We're talking about a Property Graph 8
- 27. We're talking about a Property Graph Nodes 8
- 28. We're talking about a Property Graph Nodes Relationships 8
- 29. We're talking about a Property Graph Em Joh il an knows knows Alli Tob Lar Nodes son ias knows s knows And And knows knows rea rés s knows knows knows Pet Miic Mc knows Ian er knows a a knows knows De Mic lia hae l Relationships Properties (each a key+value) + Indexes (for easy look-ups) 8
- 30. We're talking about a Property Graph 8
- 31. Datasets • FEC Data – Candidates – Committees – Contributions • Sunlight Labs – SuperPac Contributions and Expenditures • And still more could be had…
- 32. FEC Data • In 1975, Congress created the Federal Election Commission (FEC) to administer and enforce the Federal Election Campaign Act (FECA) – – The statute that governs the financing of federal elections. • The duties of the FEC, which is an independent regulatory agency, are to disclose campaign finance information
- 33. FEC Data • Detailed files about Candidates, Committees and Individual Contributions – http://www.fec.gov/finance/disclosure/ ftpdet.shtml • 10 years of data – Updated every Sunday
- 34. FEC Files • Committees – The committee master file contains one record for each committee registered with the Federal Election Commission. • Candidates – The candidate master file contains one record for each candidate who has either registered with the Federal Election Commission or appeared on a ballot list prepared by a state elections office. • Individual Contributions – The individual contributions file contains each contribution from an individual to a federal committee if the contribution was at least $200.
- 35. The Data
- 36. Data • Committee – C00000059|HALLMARK CARDS PAC|GREG SWARENS|2501 MCGEE| MD#288|KANSAS CITY|MO|64108|U|Q|UNK|M|C| | • Candidate – 4372|P80003338|OBAMA, BARACK|DEM|2012|US|P|0|I|C|C00431445| PO BOX 8102| |CHICAGO|IL|60680|NA|NA – 4373|P80003346|HONEYCHURCH, JOE|REP|2008|US|P|0|O|N| C00431155|5401 LENNOX AVE #40 B| |BAKERSFIELD|CA|93309|NA| NA – 4374|P80003353|ROMNEY, MITT|REP|2012|US|P|0|C|C|C00431171| 585 COMMERCIAL ST.| |BOSTON|MA|2109|NA|NA
- 37. Data • Individual Contributors – AJA, BAYYOGAL55283|AJA, BAY|YOG|AL|55283|LAW|MANAGE|na|na – AXEL, ANNNYNY10025|AXEL, ANN|NY|NY|10025|COLUMBIA UNIVERSITY|SOCIAL WORKER|na|na – DYSITETULSAOK74145|DYSITE|TULSA|OK|74145|N/A/WEB DESIGN|WEB DESIGN| na|na • Contributions – NASSOUR, TAMMYMALVERNPA19355|C00410266|01052011|750.0|15| AA24A9CC9439D46D7A6A|714732 – NORRIS, ELIZABETH AMALVERNPA19355|C00410266|01132011|4000.0|15| AD2BAAB7CE6234AA585D|714732 – NORRIS, JAMESMALVERNPA19355|C00410266|01042011|4000.0|15| A1B7AADE1665546EB8AF|714732 – POLLOCK, RANDALLMALVERNPA19355|C00410266|01122011|250.0|15| AC184E2EB48E14A4991F|714732
- 38. Data • SuperPAC Contributors – SA11AI Individual/Corporation|WINNING OUR FUTURE|C00507525| | Adelson|Miriam|Las Vegas|NV|Physician|Adelson Clinic|5000000|1/24/12| 5000000|SA11AI.9595|787720 • SuperPAC Expenditures – UNITED STEELWORKERS POLITICAL ACTION FUND|C00003590|N| G|"Obama, Barack"|Support|P80003338|DEM|P|0|NA|780.52|DC| 9/5/12|G|"Printing, PDQ"|Inv. # 84734 - Hard Hat Stickers - Obama Campaign|E118F9A6FA8B048EF8E4|808740 – GOVERNMENT IS NOT GOD|C00297531|N|G|"OBAMA, BARACK"| Oppose|P80003338|DEM|P|0|NA|10800| |9/5/12|G|"Envision Marketing, Inc."|postage and mail preparation|SE.10416|808752
- 39. Data Prep • Extract and Transform – Stored files on S3 – Used MortarData to run Hadoop jobs to prepare data (@MortarData) • Load – Used Neo4J BatchInserter to load • Thanks to Michael Hunger (@mesirii) • Loaded 2M+ nodes in <5 minutes
- 40. Java BatchInsert Download Use S3 data Storage Process with Hadoop/Pig Created Neo4J DB
- 41. Develop Get it perfect. Mortar Run shows you what your job Mortar creates a private will do before you run it. Apache Hadoop cluster for Mortar uses real Python, your job. Pig + Python create so NumPy, SciPy, and optimized jobs for execution. NLTK work perfectly. Read from cloud storage (S3) Process in a private, on-demand Apache Hadoop cluster Write to cloud storage (S3)
- 42. Process Text Files Using PIG • Name Standardization (Last, First -> First,Last) • Date Reversal (MMDDYYYY -> YYYYMMDD) • Create unique individual contributors with multiple contributions – 1 to Many
- 43. Why PIG/Hadoop • Cloud storage and cloud based HADOOP • Easy to use • Fast
- 44. Financial Data Example Expends s Receive s Give contribAmt: 300 Giv contribDate: 20120604 es Receives Supp Troy Smith or ts Gi s ive ve Michelle Bachman e s c Bachman for Congress Re contribAmt: 200 contribDate: 20120501 contribAmt: 830 s David Schweikert ives contribDate: 20110506 o rt G Rece pp ives Su Giv es i ves R ece Craig Stull DAVID SCHWEIKERT FOR CONGRESS contribAmt: 400 contribDate: 20110815
- 45. Data Model Expenditures Committee SUPPORTS Candidate FUNDS superPac Contributions Contribution GIVES Individual
- 46. More Details on the Data Committee Candidate committeeID candidateID committeeName candidateName committeeTreasurer SUPPORTS candidateParty committeeCity candidateElectionYear committeeState candidateOfficeState committeeZip candidateOffice FUNDS Contribution Donor indivName commID indivCity contribDate GIVES indivState contribAmt indivZip contribType idivEmp indivOccupation
- 47. More Details on the Data Expenditures commID Committee contribDate SPENDS_MONEY contribAmt contribType committeeID committeeName committeeTreasurer committeeCity committeeState committeeZip SUPERPAC_GIVES superPac Contributions commID donorName donorCity donorState donorZip donorAmt donorDate
- 48. Cypher • Developed for and by Neo4J
- 49. Can I Draw Patterns A B C
- 50. Sure, with ASCII Art! A B C A - - > B - - > C, A - - > C
- 51. Sure, with ASCII Art! A B C A-->B-->C<--A
- 52. Directed Relationship A B (A) - - > (B)
- 53. Labeled Directed Relationship Gives A B (A) – [:Gives] - > (B)
- 54. Variable Length Path A B A B A B (A) – [*] - > (B)
- 55. Let’s take a look….
- 56. GitHub for Data and Code https://github.com/akollegger/ FEC_GRAPH 34
- 57. All Candidates in Minnesota • START n=node:candidates(‘candidateOfficeSta te:MN*') • where nreturn n
- 58. Simple Contribution • start a = node(5058) • match p = a-[:RECEIVES]-(b)<- [:GAVE]-(c) • where 1=1 • and b.contribAmt > 1 • return c 36
- 59. Bachman Contributions // Bachman Contributions start a = node(3946) match p = a-[:RECEIVES]-(b) where 1=1 and b.contribAmt > 1000 return b
- 60. Real World Examples • Cash pours into state campaigns from afar – http://www.startribune.com/printarticle/? id=162672646 – A Star Tribune analysis of her (Bachmann’s) most recent financial reports shows that up to 80 percent of her reported contributions came from individuals outside Minnesota
- 61. Bachman Contributions • start a = node(3946) • match p = a-[:RECEIVES]-(b)<- [:GAVE]-(c) • where 1=1 • and (c.INDIVSTATE? = "MN") • and b.contribAmt > 1 • return sum(b.contribAmt) 39
- 62. Julie Waddle START person = node(191468) MATCH person-[*]->c<-[*]-d return c.contribAmt, c.contribDate, d.name 40
- 63. SuperPac Examples START n=node:committees('commName: 1911*') MATCH n-[r:SUPERPACEXPEND]->c RETURN c.PURPOSE, count(c.PURPOSE) ORDER BY count(c.PURPOSE) DESC 41
- 64. More SuperPac Examples START n=node:committees('commName:*') MATCH n-[r:SUPERPACEXPEND]->c WHERE c.PURPOSE! <> "ZOO" RETURN c.PURPOSE, count(c.PURPOSE) ORDER BY count(c.PURPOSE) DESC START n=node:committees('commName:*') MATCH n-[r:SUPERPACEXPEND]->c WHERE c.PURPOSE! = "Rental Van" RETURN c.PURPOSE, count(c.PURPOSE), c.CANDIDATE, sum(c.expendAmt) ORDER BY count(c.PURPOSE) DESC 42
- 65. Other Resources • https://github.com/ akollegger/FEC_GRAPH 43
- 66. Thank you! @davefauth
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: \n
- #61: \n
- #62: \n
- #63: \n
- #64: \n
- #65: \n
- #66: \n
- #67: \n
- #68: \n
- #69: \n
- #70: \n
- #71: \n
- #72: \n
- #73: \n
- #74: \n
- #75: \n
- #76: \n
- #77: \n
- #78: \n
- #79: \n
- #80: \n
- #81: \n
- #82: \n
- #83: \n
- #84: \n