MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB

MongoDB Days Silicon Valley
Winning the Dreamforce
Hackathon with MongoDB

db.mongoDaysSV.insert(
{
“init”: “Welcome Message”,
“author”: “Greg Deeds”,
“twitter”: “@TEGBIZ”,
setInterval: 162000
}
);

RAPID PROTOTYPING
with MongoDB

You want to change
the game?
You have 24 hours...

Two True Tales Of
MongoDB
MongoDB & Force // GR
• API driven rapid prototyping
• HEROKU CONTAINERIZATION
• MASS IMPORT DATA SCIENCE
DRONEY EXPRESS // MongoDB your range
• Geospatial & Autonomous

Winning with MongoDB
• Developing Rapid Prototypes
• Winning Hackathons with MongoDB
• Drones / MongoDB {Range Management}

MongoDB
or
stay in the RDBMS Matrix

Give Up The Old And Win
• Traditional DevOps Practitioner moved to Micro-services
{Microsoft .NET & SQL Server}
{Oracle APEX-PL/SQL & Oracle Server}
{Any Linux... MySQL/POSTGRES}
• Teamwork to overworked evolutions
• My SQL Dreams never materialized - {WHATS NEXT!}
• AGILITY TRAIN YOUR MIND AS A LIFESTYLE

MY BACKGROUND
• Started in 1979 {Ti99/Tape Deck}
• Wrote ﬁrst database on Commodore
• BBS Author/ISP {One of the young ones}
• Multimedia Producer // Software Engineer
• Worked for corporations to startups
{ Military / Federal / Education }
• The Writing is on the Wall - MongoDB Time!

Languages & Coding
{Berkeley Logo}

Languages & Coding
{Learning to Communicate}

Languages & Coding
{Standing on the Shoulders of Giants
to Communicate}

MongoDB
Queries Support Projection!

PROBLEM: The data is available but in various formats,
not easy to use and is changing
• Accessing your government should not be data
science or outside of SalesForce.com
• PLATFORM HEROKU - {the app processing grid}

Our target user ranges from the socially conscious individual to Publicly Traded
Companies and Governments. Our comprehensive tool will allow…
• Grassroots Organizations, Associations
• Publicly Traded Companies
• Lobbyists, PACs and Governments
Manage information
• Up-to-Date Congressional Legislator Directories
• Congressional / State Legislation Tracking
• Easy Meetings Management
• Bill and Amendments Tracking
• Committees and more
We are most proud of the tool because it allows both individuals and
organizations to understand the landscape of our Government, politics,
and social activism.

ForceGR / SPEC
• FORCE.COM APP Generator
• APEX & PL/SQL Classes
• Restful NODEJS HEROKU
• Mongo for Import, Reduce and Production

Data Sets Matter
• Variety Of Formats
• YAML // XML // JSON // CSV // TAB
• Normalized only for XML / DTD files.
• Multiple websites crawled/parsed
{Using RSYNC || PHP To Bulk Load Objects}
• Aggregate Pipeline
• Build Words / Terms from triples

• Semantic to understand {What you say}
• Latent Semantic to understand {What you mean}
• MongoDB // Provides both when used in pipeline
with other word bag or terminology generators
• The results are we find correlation in unnormalized
data
Data Miner Mongo DB

• Monitor the query in MongoDB reactively
{Check to see how the indexes need to improve with a statistics
collection of performance and counters for the API}
• Compound indexes need to be built on geojson first
then other fields when used
• Consider using elastic search to boost the Hackathon
Team to victory if free text is slow
Use Conditional Indexes
{add Agility_new ways working with this data?}

• Find source files over APIs //
{ to maintain high speed with no throttles. Hackathon
bandwidth is varied sometimes when you need it. It gets
worse with time }
• Use Old School Shell Tools //
{ NodeJS is not ideal for all challenges }
• PHP
• PERL
• PyThon
• RSYNC
MISSION: Data Capture

• The focus of TEG’s FORCEGR tool is to
present information into SalesForce users
Desktop
• Then to make decisions about responses to
these decisions
• Further guiding the NodeJS learning engine
APEX CLASSES / INTERACTIONS

question =
(my_department, has, what)
knowledge = (
(I, own, my_apartment),
(my_department, has,
my_computer),
(my_department, has, my_vote),
(my_department, is_in,
Philadelphia)
)

‘$push’
‘$pop’
‘$unset’
‘$set’
bye bye SQL...
Sew the Dataset

• .NET/Oracle APEX/PLSQL is heavy Spiral DEVOPS
• The Browser Wars- We Won {HTML5/Javascript Rocks}
• SUBVERSION TO GIT DEVOPS TO NEW CHATOPS
{Your Dated If You Can’t GIT With It…}
• Shifts in web computer science
• Multitouch surface the GPU and JavaScript are the
next big shift
Recognizing Shift

• Practice writing specifications for NOSQL
• Use New tools to trap ideas
• CHECK YOUR ROADMAP?
{ are you on course & time? }
• Gain momentum by solving problems quicker
• GIT more done { Practice like you play the game }
• Organization to Reproduce Results
{use automated ‘Agents’ when possible & makes sense}
{containerize your app with docker or Heroku}
Next Generation DevOps

question = (my_department, has, what)
knowledge = (
(I, own, my_apartment),
(my_department, has, my_computer),
(my_department, has, my_vote),
(my_department, is_in, Philadelphia)
)
for each statement in knowledge {
if ((statement.subject == question.subject
or question.subject == what) {
and (statement.predicate == question.predicate
or question.predicate == what)
and (statement.object == question.object
or question.object == what))
call FoundAnswer(statement)
}
}
Output:
Answer: my_department has my_computer
Answer: my_department has my_vote
Pseudocode for Question-
Answering Logic in NodeJS

• Ask yourself on each project freshly
{What else is aggregation good for?}
• HINT: You can use the $out pipeline
operator and it will create or replace the
collection it spits out, after the results
are ready....
Aggregation Tips
{ Use the Pipeline, Luke! }

if (! Links.ﬁnd().count()) {
// init data
var nodeNames = [];
var data = ['Harry,Sally,1.2',
'Rubio,Mario,1.3',
'Obama,Barrack,0.2',
'Thompson,Fred,0.5',
'Peter,Johan,0.7'];
_.each(data, function (val) {
var comp = val.split(',');
Links.insert({ source: comp[0], target: comp[1], compute: comp[2] });
nodeNames.push(comp[0]);
nodeNames.push(comp[1]);
});
// remove duplicates
nodeNames = _.uniq(nodeNames);
_.each(nodeNames, function (name) { Things.insert({ _id: name }); });
}
Finesse by using new tools

VICTORY
It never lasts long enough....

• HTML5 & WebGL ODM Tools
• MongoDB Powering Geoserver Raster
• Realtime 3-D Model Scenes In A Browser
Hibernate ORM is Dead
{long live MongoDB ODM}

Hibernate ORM VS MongoDB ODM
• Hibernate Maps Java Objects To RDBMS And Is
Proven Not Scalable
• RDBMS load times for a single object graph
(Hibernate/SQL Server) from 3 to 4 seconds
• Relational model is not suited for complex object
graphs.
• Even with massive parallel processing is not
economical
Hibernate Search
• Cannot Beat Elastic-search & MongoDB In
Response Speed
Business Opportunity

• ORM Hibernate JBOSS - Object/Relational Model
{storage has reached the end of life period…}
• MongoDB is the best object store on the market today
• Hibernate is complex when the world needs BSON
• MongoDB is Built For Replication
• Hibernate Replacement Is An Opportunity.
Market Need {MongoDB Sensor ODM}

Project Requirements
• Open Source/No Internet/Limited WiFI
• Runs on variety of low cost hardware
• Auto-deploys Via Docker On IoT Appliance
• Plugin To Existing GeoServer/ESRI/GRASS
• Replace Hibernate With MongoDB
Droney Express
{Range management System}

• Interchange GeoJSON set of Shapes & Waypoints
• RasberryPi Flight Computer running 2d Queries
• Objects Linked To External GIS Engine (GRASS/ESRI)
• AutoPilot Configuration In Flight/Over-The-Air
• MongoDB Monitors Situational Awareness Of
Missions
• GIS Query Results Stream As BSON Objects
What is a Mission?

Geographic Forensics
{sensor data store/search}

The work of Droney Express
• Planned Geofence Scanning
• 2D/3D indexed sensor data
• GIS & MongoDB Is Powerful
• Integrated GIS Remote Monitoring
• Connect Using DDP To Sync Collections Adhoc

Async Data Sync -> Use Oplog
For Synchronization
Serve GIS markup with pretty
maps using Geoserver...
MongoDB is perfect for
Sensors // sorry Oracle

Flight Plan For Best Routes
GIS Engines Accelerated By
MonogDB Can Handle
more ecological range management
sensors
faster coordinated ground and drone
assets

GEOJSON / MONGODB 2d
DRAINAGE MODELING

GIS ENGINE & PLUGIN
Droney Express Flight Planner
Least Cost Power & Route Calculator Export Tools For 3DR Ardupilot.

Replica Sets Are Ideal For
Network Segregation
HQ is Always Primary - Drones Always Secondary
Force Constant State of Disaster Recovery - Dedup
Drones Are Small Autonomous Collection Vehicles
Eventual Consistency In Loosely Coupled Networks
MongoDB Makes Life Easy
Secondaries DeDup Secondaries To Update HQ GIS
Backends For Mission Planning & Sensor Datasets

Droney Heartbeatmodeled from mongoDB University
Airborne MongoDB Prevent Election Of New Primary
Droney Flight Computer Always Have Priority Of 0
They Never Get Elected To Primary
[Priority, Optime, Connections]
Sync Using Oplog To HQ After Weeks In Field
Network Environment Spotty For MongoDB Servers

Statement Replication
...Getting The Data To HQ
Droney Acquires Forensic Sensor Data - ‘A Data Silo’
Facilitates Greater Independence
Replicate Based Oplog For Forensic Time-coding
Member Replica Sync Stored Records Using Oplog
Use Oplog To Journal AutoPilot For Flight Review
MongoDB Secondaries Apply Unique ID Vs Primary
Keeps Mobile Workers From Overwriting Datasets

Cheap IoT Devices MongoDB 32bit
Flight Computers & Range Sensors With Embedded MongoDB
In Flight Computers Are Low Power/LINUX Threaded
Sensors Streaming Data To/From MongoDB
Autopilot Flight Recorder Is MongoDB/NodeJS
IoT SD Cards - Ideal For Circular Buffers In Flight
Circular Buffers Makes Room For New Documents
Overwrite The Oldest Documents In A Collection

Final Thought..
NO CAMPING … KEEP CLIMBING!
CONSIDER THE FUTURE OF YOUR TIME
STUDY NEXT GENERATION DEVOPS
SAY “WHY CANT WE FIRST”
SHARPEN UP REGULARLY
TEST YOUR SKILLS OR TEACH A SKILL

Greg Deeds
Technology Exploration Group
SALES@TEGbiz.com
www.TEGbiz.com
@TEGbiz
@DRONEYEXPRESS
@3dVRviewers

MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB

More Related Content

What's hot

Similar to MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB

More from MongoDB

Recently uploaded

MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB