MarkLogic 8: Developer Experience

(1)

MarkLogic 8: Developer Experience

Justin Makeig November 2014

(2)

MarkLogic 8 Feature Presentations

Topics Product Manager

Developer Experience: Samplestack and Reference Architecture Kasey Alderete

Developer Experience: Node.js and Java Client APIs, Server-Side

JavaScript, and Native JSON

Justin Makeig

REST Management API, Flexible Replication, Sizing, and Reference Hardware Architectures

Caio Milani

Bitemporal Jim Clark

(3)

DEVELOPER EXPERIENCE

(4)

New and Enhanced features in MarkLogic 8

 Samplestack: Reference architecture instance

 Client APIs: Extensible drivers for Java and Node.js

 Open development: Transparency and responsiveness with GitHub-first  Server-Side JavaScript: Run JavaScript close to data

(5)

Data Services

 Documents, collections, elements

 JSON/XML over HTTP

Business Services

 Resources (Customer, Approval, etc.)

 JSON over HTTP

Database

Middleware

User Interface

Business rules, domain model, integration

Data views, user workflow

Persistent state, stored procedures _Sec

ur it y, M oni tor ing, C onf ig Mg mt

(6)

(7)

New and Enhanced features in MarkLogic 8

 Open development: Transparency and responsiveness with GitHub-first

 Server-side JavaScript: Not XQuery

(8)

REST Client API

HTTP Data Services

{Java, Node} Client API

{JavaScript, XQuery} Built-ins

Extensions

Application Logic

(10)

Java Client API

NoSQL agility in a pure Java interface

 Faster development and less custom code with out-of-the-box data management, search, and alerting

 Pure Java query builder and conveniences for POJOs, JSON, XML, and binary I/O

 Built-in extensibility for moving performance-critical code to the database

 Always open source and developed on GitHub

Participate. Contribute. Fork it.

(11)

Deploy in every environment

 Increase flexibility by reusing existing skills, tools

 Minimize integration costs with a pure Java interface  Maximize performance by bringing code to the data  Scale up (or down) without modifying application code  Build, test, instrument, debug with standard tools

(12)

Simpler data integration

 Reduce custom code for transactions, security, marshalling, orchestration  Increase flexibility by mixing POJOs, JSON, XML, and triples

 React more quickly to change by using data in its natural format with less ETL  Maximize performance by bringing code to the data

(13)

(14)

Bulk writes

JacksonHandle handle = new JacksonHandle();

GenericDocumentManager docMgr = client.newDocumentManager(); DocumentWriteSet writeSet = docMgr.newWriteSet();

for (JsonNode json : myCollection) { handle.set(json);

writeSet.add("/" + i + ".json", meta, handle); if (i % BATCH_SIZE == 0) {

docMgr.write(writeSet);

System.out.println("Wrote batch"); writeSet.clear();

} }

(15)

POJO façade

 Manage and query POJOs  Inspired by Spring Data

 “Cheap and cheerful”: Not a Hibernate/JPA substitute

PojoRepository<Product> repo =

client.newPojoRepository(Product.class, Long.class);

PojoQueryBuilder qb = productRepo.getQueryBuilder(Product.class); QueryDefinition query =

qb.containerQuery("company").value("name", prod1.getName());

for (Product result : productRepo.search(query, 1)) { … // process each product

(16)

Annotation-based range index creation for POJOs

 Deployment automation lifecycle

1. Annotate domain class getters

2. Run included GenerateIndexConfig to generate config

3. Post to the Management REST API

@PathIndexProperty(scalarType = ScalarType.DOUBLE)

public Double getBalance() {

return balance; }

(17)

Eval and invoke

 Eval ad hoc code, invoke server-side modules  JavaScript or XQuery

 Type marshalling

 Sharp tool: Lead with resource, transformation extensions

ServerEvaluationCall exp = client.newServerEval()

.javascript(javascript) // String of Server-side JavaScript

.addVariable("percent", 0.08);

(18)

Java Client API or XCC?

 XCC is not going away: Hundreds of customer apps, mlcp, Hadoop Connector, .NET, etc.

 Start with the Java Client API: Easy to get going, built-in best practices, extensible  Eval/invoke narrows the functionality gap.

(19)

Node.js Client API

Enterprise NoSQL database for Node.js applications

 Focus on application features rather than plumbing with out-of-the-box search,

transactions, aggregates, alerting, geospatial, and more

 Move faster to production with proven reliability at scale

 Maximize performance and flexibility—bringing code to the data

 Enable modern end-to-end JavaScript development

 Always open source on GitHub

Participate. Contribute. Fork it.

(20)

Straightforward data integration

 React faster to change, using data in its most natural form with less ETL: JSON, XML, RDF, text, binary

 Transactional multi-document updates ensure consistency  Async, promises, streams ensure seamlessness with Node

 Reduce data movement, duplication by moving code to the data and invoking from Node

(21)

Deploy in every environment

 JavaScript is everywhere: Reuse skills, tools, investments

 Node.js as standard middleware for connecting JSON services over HTTP with JavaScript

 Scale up (or down) without modifying application code  Build, test, instrument, debug with standard tools

(22)

What is Node.js and why is it important?

 Scripting environment for network services with JavaScript

 Event loop: Single thread, non-blocking I/O, and asynchronous events  “N” in MEAN: JavaScript-JSON throughout the stack

(23)

Key concepts

 Promises: Humane async chaining and error handling (using Bluebird)  Streams: Observable data flow… think UNIX pipes

(24)

(25)

Quick Quiz: What’s the order of the function calls?

function A(callback) { … } function B(callback) { … } function C(stuffFromA) { … } function D(thingsFromB) { … } A(C); B(D);

(26)

var marklogic = require('marklogic');

var conn = require('./env.js').connection; // Host and auth details

var db = marklogic.createDatabaseClient(conn);

var q = marklogic.queryBuilder; db.documents.query(

q.where(

q.collection('countries'), q.value('region', 'Africa'), q.or( … )

) )

.result(function(documents) {

documents.forEach(function(document) { … }) });

(27)

(28)

New and Enhanced features in MarkLogic 8

 Server-Side JavaScript: Not XQuery

(29)

Server-Side JavaScript

Front End Middle Tier Database Layer

 Run code near the data for unparalleled power, efficiency

 Build applications faster from a growing pool of skills, tools

 Reduce risk with proven performance and reliability

 Decrease brittle ETL and lost fidelity and functionality from JSON data conversions

 Pair with Node.js to ease full-stack JavaScript development

+

(30)

Intelligent data layer

 Maximize performance by bringing code to the data

 Parallel search and aggregates minimize data movement  Built-in HTTP app server simplifies application architecture  Reuse JSON data models, JavaScript code across tiers  Async tasks, batch processing increase flexibility

(31)

Better answers from today’s data

 Model and manipulate documents, relationships, and metadata—combining JSON, XML, RDF, text, and binary

 Unified JavaScript interface for all indexes, data formats

– Text search, semantic inference, aggregates, geospatial, alerting  Real-time consistency when milliseconds count

(32)

Simpler data integration with JavaScript

 Transactional multi-document updates ensure consistency

 React more quickly to change by using data in its natural format with less ETL  Rich real-time indexes reduce custom code

 Ecosystem of data processing libraries ease development  Streamline ETL with JavaScript built for JSON data

(33)

REST Client API

HTTP Data Services

{Java, Node} Client API

{JavaScript, XQuery} Built-ins

Extensions

Application Logic

Application

MarkLogic

(34)

(35)

“Hello, world!”

var q = cts.andQuery([cts.wordQuery(…), …]);

var itr = subsequence(cts.search(q, …), 1, 10); for(var result of itr) {

var obj = result.toObject(); …

}

(36)

Built-in Types

 Value: .toObject() and .valueOf()

 ValueIterator: Lazy loaded “sequences”, ES6 iterator

for(var item of iterator) { … } Eager? iterator.toArray()

 Node: Document, ObjectNode, XMLNode, etc.

cts.doc('/thundersnow.json') instanceof Node; // true

(37)

Nodes vs. Objects

 Nodes: What’s in the database

– Immutable (just like XQuery)

– JSON (object, array, number, …), XML (element, attribute, …), binary, text

 Objects: What’s in your code

– Mutable: obj.fullName = "Nigel Tufnel"

(38)

Nodes vs. Objects

fn.collection() // ValueIterator

.next() // Iterate

.value // Document node

//.root // ObjectNode (not required)

(39)

Updates

declareUpdate();

for(var item of fn.collection("accounts")) {

var obj = item.toObject();

obj.balance = obj.balance * 1.05;

var collections = xdmp.documentGetCollections(uri); xdmp.documentInsert(item.nodeUri, obj,

xdmp.defaultPermissions(), collections); }

(40)

Namespaces and modules

 Global namespaces: xdmp, cts, sem, etc.

 CommonJS-style modules: module.exports, require()  Same path resolution, precedence as XQuery (not Node.js)  Import XQuery, employ as JavaScript

– Public functions and variables – Type mapping

– Automatic camelCase conversion:

(41)

Server-Side JavaScript != Node.js

 Complementary, but very different

 Both use V8, but are separate environments, processes  Share models, libraries, and patterns between them

MarkLogic Server-Side JavaScript Node.js

Sync interface, asyc below Async throughout xdmp.*request|response* and http.* require('http')

(42)

Coverage and Performance

 Comprehensive coverage of built-ins

 Import existing XQuery modules (admin, search, etc.)  V8 is fast…for computation on an E-node

(43)

(44)

(45)

QUERY LAYER _SQL

SPARQL XQUERY

EVALUATION LAYER EVALUATOR QUERY CACHE

BROADCASTER | AGGREGATOR

DATA LAYER _{TRANSACTION CONTROLLER}

INDEXES

COMPRESSED STORAGE

DATA CACHE TRANSACTION JOURNAL JAVASCRIPT

(46)

New and Enhanced features in MarkLogic 8

 Server-side JavaScript: Not XQuery

(47)

JSON

Unified indexing and query for today’s web and SOA data

 Speed up development with powerful built-in search, transformation, and alerting capabilities designed for JSON

 Reduce lost fidelity and functionality from data model translations and brittle ETL

 Simplify architecture with data, metadata, and relationships managed consistently and securely together

 Ease modern, end-to-end JavaScript development 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 { "_id": 1, "name": { "MarkLogic" }, "supports" : [ { "datatype": "XML", "year": 2003 }, { "datatype": "JSON", "year": 2014 } ] }

(48)

Better answers from today’s data

 Mix and match the most appropriate data formats without costly up-front schemas

– JSON: Data structures and hierarchies – XML: Markup and rich text

– RDF Triples: Facts and relationships

 Decrease development time and governance costs with unified management, indexing, security across formats

(49)

Straightforward data integration

 Reduce the cost of translations and duplication working with data from multiple sources, likely already JSON

 React more quickly to change by using data in its natural format with less ETL  Reduce risk and governance costs with fewer data silos

(50)

JSON

XML

JavaScript

︎

XQuery

︎

● ◐

(51)

Boolean

isActive affiliation Null

Number scores Number scores Number scores Text name Array scores Object unnamed Document unnamed true "Ol…" 88 67 73

JSON data model

{ "name": "Oliver", "scores": [88, 67, 73], "isActive": true, "affiliation": null }

(52)

Integrated

 Full-text, value, scalar, geo, triple indexing

– Strong typing (no tokenization) for numbers, booleans, null – Support for GeoJSON and ArcGIS points

– Dates and Arrays just work®

 XPath and XQuery

doc.xpath('/node()[some $n in friends/name satisfies starts-with($n, "A")]');

 Seamless in JavaScript

(53)

JSON or XML?

JSON XML

Data structures Markup

Types: string, number, boolean, null Text: Language, mixed content

JavaScript XQuery, XPath, XSLT, Schema, …

(54)

Migrating from the existing JSON façade

 Sample upgrade script ships with 8.0-1

 Additional Code and configuration changes (e.g. xdmp.to-json())

XML

(55)