Challenge to add OpenStack
API Validation Framework
2013.05.31
Ken’ichi Ohmichi NEC-Soft, Ltd.
Page 2 NEC Corporation 2013
Agenda
▌
OpenStack
▌
OpenStack community development
▌
OpenStack functions through RESTful API
▌
Issues of RESTful APIs
▌
Fail-Fast architecture
▌
API validation framework
▌
Development process
OpenStack
▌ The Open Source Cloud Operating System
▌ Controls virtual machines, storage/networking resources.
▌ Consists of multiple functional components.
Compute: Nova VM image: Glance Identity: Keystone
Networking: Quantum (will be renamed) Block storage: Cinder
Object Storage: Swift Dashboard: Horizon Metering: Ceilometer Orchestration: Heat
OpenStack community development
▌
The community is very active.
OpenStack community consists of many developers, users and
operators.
They are cooperating each other on mailing-list, IRC and development
tools such as Gerrit.
The community releases new version of OpenStack every six months,
OpenStack community development
▌
The development routine
1. Submit a bug-report / blueprint to OpenStack Launchpad.
2. “git clone” necessary source code from OpenStack github.
3. Modify the source code as you like.
4. “git commit” with the id of bug-report / blueprint.
5. “git review” for posting a patch into Gerrit system.
6. (Reviewers) review the patch and submit their feedbacks with score on Gerrit.
Score +2: Approve
Score +1: Looks good do me, but someone else must approve Score -1: I would prefer that you didn’t merge this.
Score -2: Do not merge
7. Apply reviewer’s feedback to your patch, and retry “git review” for reposting it.
8. (Reviewers) re-review the patch. When core reviewer marks it as +2, the patch is merged to OpenStack github. Congratulations☺
OpenStack functions through RESTful API
▌
Control functions through RESTful API directly.
EX) We can create a virtual machine by sending API request with
following parameters to http://<openstack>:8774/<tenant-id>/servers with
POST method. The parameters can be specified with JSON/XML format.
{ "server": { "flavorRef": "1", "imageRef": "70a599e0-31e7-49b7-b260-868f441e862b", "name": "new-server-01" }
▌
Control functions
with dashboard or
command line.
▌
These tools uses
RESTful API in the
background.
Create vm bypushing here.
Page 7 NEC Corporation 2013
Issue/Solution of RESTful APIs
▌
Issue
Not all API parameters are completely validated, so some
API
operations execute without parameter validations.
▌
Solution
By validating every API parameters just after receiving API request, it
makes better OpenStack quality.
Fail-Fast Architecture
▌ Fail-Fast system does not retry/cleanup if something wrong happens.
Ex) If something wrong happens after adding a DB record during one API
operation, it does not remove the added record. As the result, incomplete record remains in DB.
▌ The antonym is Fail-Safe.
▌ The implementation cost of Fail-Safe tends to higher:
“What should be cleaned up when error handling?”
“What should it do when double errors (something wrong in error handling)?”
▌ On Fail-Fast system is easy-debugging because developers can know what happens easily.
Op-01 Op-02 Op-03 Record-01 add
Fail-Fast
remains Op-01 Op-02 Op-03 Record-01 addFail-Safe
removed delPage 9 NEC Corporation 2013
API validation framework: Goals
▌I have proposed API validation framework to OpenStack community
for comprehensive validation.
▌This framework goals are:
To validate every API parameter.
To return an error response before API operation, if invalid API
parameter.
To unify the error message format of the response, if the same
cause.
ex) “.. is not string.", ".. is not integer."
▌By comprehensive validation just after receiving API request, it
makes better OpenStack quality because it can reduce incomplete
situations like incomplete record remaining.
Page 10 NEC Corporation 2013
API validation framework: Validation Point
(discussing now)▌The API validations are executed before each API method.
▌The proposed framework is implemented with jsonschema library.
▌This framework can validate requests of both XML and JSON, because jsonschema handles “dict” data after JSON/XML deserialization.
I heard often “Can this framework handle XML request?” due to its library
name.
The answer is “Yes, it can.”
Nova
Client request DeserializationJSON/XML Router
(2) Execute the selected validation (1) Select a validation and a method corresponding to API (3) Execute the selected method API Validation API Validation API Validation API method API method API method
Page 11 NEC Corporation 2013
API validation framework: API Schema
(discussing now)▌
“API schema” = API parameter data format
.
Ex) The API schema of the “Change Password” API.
{
'type' : 'object', 'properties' : {
'changePassword': { 'properties' : {
'adminPass': {'type': 'string', 'required': True}, }
} } }
▌Need many API schema definitions because OpenStack has many APIs (hundreds).
Page 12 NEC Corporation 2013
API validation framework: Merits
▌In addition to the goals, there are two merits of the framework. ▌Clarification of API Parameter Definitions
The type and acceptable range and length of each parameter are clarified in the API
schema definitions.
By defining the schemas of every API, developers will be able to create applications
more easily using Nova APIs.
▌Cleaned-up Code
The amount of Nova code can be reduced by merging validations and error response
methods.
Ex) the “min_count” parameter of “create a virtual machine instance” API
try:
min_count = int(str(min_count)) except ValueError:
msg = _('min_count must be an integer value') raise exc.HTTPBadRequest(explanation=msg) if min_count < 1:
msg = _('min_count must be > 0')
raise exc.HTTPBadRequest(explanation=msg)
Development process
▌ March, 2013
Created some prototype for Nova.
Proposed the prototype to OpenStack community. Could listen positive responses.
▌ April, 2013
Had a session of OpenStack Design Summit for this framework in Portland, Ore. Could make consensus for its purpose.
Could know it is good that the first target of this framework is Nova v3 API. Nova v3 API will be implemented for Havana, and compatibility issues do not
Page 14 NEC Corporation 2013
Schedule for Havana
▌Now we are discussing what is best validation mechanism.
We need more time for consensus.
▌We have 4 months until Havana-3 meaning deadline for new code.
Make Framework Core Patches.
Make API schema Definition Patches.
• This is very hard work, because Nova has many APIs. • Merge by middle of August.
we May
Apr June July Aug Sep Oct
AD 2013
h3 h1 h2 rc1 rc2 rc3 Havana Framework coreDeadline for new code
▌
OpenStack community development is hard work,
but contribution to the community is very interesting
because the community consists of skillful
developers and they give good questions, advices,
and answers.
▌
I am glad if many people are into OpenStack
community.
Page 17 NEC Corporation 2013
Related URLs
▌Blueprint “Nova API Validation Framework”
https://blueprints.launchpad.net/nova/+spec/nova-api-validation-fw
▌Framework core patch for Oslo
https://review.openstack.org/#/c/25884/
▌Framework core for Nova
https://review.openstack.org/#/c/25358/
▌Nova v3 API
https://blueprints.launchpad.net/nova/+spec/nova-v3-api
