Assembling Web Development Environments with Perforce. Stephen Vance Perforce User Conference 2001

Assembling Web

Development Environments

with Perforce

Stephen Vance

Once Upon A Time …

n Experiences from

G Consulting for Web companies

G Involvement with Internet Start-up

G Problem-solving from personal interest

n Internet start-up is main storyline. n Of course, names changed.

The Characters

n Me: “I can’t let chaos ruin us”

n The CFO: “We’re on a tight venture budget” n The CTO: “I can develop that in two hacks”

n The Marketing Guy: “I can’t live without my Mac” n The Sub-contractor: “We do what we do”

The Case for SCM in Web

Development

n Many disciplines converging in one medium q Programmers

q HTML designers

q Advertising executives q Marketers

n Many skill levels from novice to expert

n Large-scale sites are still driven by custom-coded

In The Beginning …

n There was the void.

G a.k.a. a start-up with funding and no developers

n And then he said, “Let them have options”

and product development commenced.

n Against better judgment we started hacking

Aaaarrrrgh!

n And as one might expect, some code was

irretrievably flattened at 3:00 a.m. by two people working on the same file.

n After some “discussions” with the CTO and

The Web Site Conundrum

n Each developer has his own workspace n But there is still only one Web server

n Although source changes are safe against

multiple authors, there is only one testing environment

Virtual Hosts

n IP- or name-based

n IP-based virtual hosting has shortcomings

n Name-based virtual hosting addresses these

Name-based Virtual Hosts

sites based on name in Host header

n DNS wildcards avoid need

for DNS reconfiguration

n Apache configuration

handles the rest

GET / HTTP/1.1

Host: www.vance.com

*.dev.vance.com. CNAME 86400 devhost.vance.com.

NameVirtualHost * <VirtualHost *> DocumentRoot /usr/local/apache/htdocs ServerName www.vance.com </VirtualHost> <VirtualHost *> DocumentRoot /usr/local/dev/site1/htdocs ServerName site1.dev.vance.com </VirtualHost>

Visualizing Virtual Hosts

http://site1.dev.vance.com/test.html

*.dev.vance.com. CNAME 86400 devhost.vance.com. devhost.vance.com. A 86400 192.168.1.1 DNS devhost.vance.com site1.dev.vance.com? Apache (devhost.vance.com) GET /test.html HTTP/1.1 Host: site1.dev.vance.com /usr/local/dev/site1/htdocs/test.html Gets file Returns page Displays page

Enter The Marketing Guy

n Technically savvy, but non-developer n Rejected disk sharing technologies n No Mac solution for non-developers n Used Windows, but

G Hated it

G Required logging in to sync virtual host

P4Web to the Rescue

n P4Web is a Perforce-specialized Web server n Provides GUI on any platform with a browser n Runs on almost any platform supported by

Perforce

n Allows management and viewing of Web

P4Web Configurations

The Next Challenge

n The Marketing Guy is happy.

n Then he announces that he has hired a

sub-contractor to do the Web design.

n Quite reasonably, the sub-contractor doesn’t

feel he needs to learn Perforce to fulfill the contract.

Easy FTP Integration

n Web design tools support ftp

n Put sub-contractor work on its own branch n Create a client with “allwrite” option

n Create virtual host with document root of <client

root>/htdocs

n Tell sub-contractor to set ftp location to <client

root>/htdocs

n Populate /cgi-bin if necessary

n Use Tech Note #2 techniques when sub-contractor

Improved Integration Using P4FTP

n P4FTP is Perforce FTP server where session

is a submission

n Isolate work to a branch

n Excellent solution, but two drawbacks

G Content only transiently populates disk, so not fit

to work with virtual host for viewing

G Each ftp user consumes a Perforce license and

Where We Are

n Virtual hosting allows multiple Web servers

on minimal hardware

n P4Web allows browser-based use of

Perforce client functionality with viewing of static content

n FTP and P4FTP allow smooth integration

with Web design tools

n However, we still have to sync a Unix client to

Viewing Content with WebKeeper

n What?

q Apache module allowing direct viewing of content from Perforce depot

q Open source project in the Perforce public depot q Recently enhanced

n Who?

q Originally authored through Perforce q I was recently designated curator n Where?

q Info at

http://public.perforce.com/public/perforce/webkeeper/index. html

How WebKeeper Works

WebKeeper Installation

n Instructions in README.WEBKEEP in Perforce

public depot

n Builds either statically or as loadable module n Builds either using full Apache source tree or

using apxs with Red Hat apache-devel.

n Built and tested on FreeBSD and Linux. n Other platforms on request

n Plan to add binary loadable modules for

Basic WebKeeper Configuration

maps Web root to Perforce depot

n IfModule makes configuration

Why Do We Need More?

n In two words, “Dynamic Content” q Text-based

n Server-side includes (SSI)

n PHP n Perl n ASP n JSP n Server-processed XML and XSLT q Binary n C/C++ CGI n EJB n Servlets

Text-based Dynamic Content

gives a client spec

n WebKeepSync tells

WebKeeper to sync the files to the client for Apache

n WebKeepRefresh

(added after paper) always refreshes include directory

Adding CGI Support

ServerAdmin [email protected] ServerName site1.dev.vance.com

DocumentRoot /usr/local/dev/steve/htdocs

ScriptAlias /cgi-bin /usr/local/dev/steve/cgi-bin <IfModule mod_webkeep.c>

WebKeepPort perforce.vance.com:1666 WebKeepUser steve

WebKeepDirectoryIndex index.html

WebKeepAlias /cgi-bin/ //depot/dev/steve/cgi-bin/ WebKeepAlias / //depot/dev/steve/htdocs/ WebKeepClient my_client WebKeepSync On </IfModule> </VirtualHost> n DocumentRoot re-introduced for file access n ScriptAlias tells Apache which content to treat as CGI scripts n New WebKeepAlias tells WebKeeper how to populate the CGI directory

n WebKeeper handles aliases in the order presented in the configuration, so the CGI alias must come first

Compiled Web Elements

n Web elements that

G Require compilation before being sent to browser G Require compilation before executing on server

n Examples

G Java applets G C/C++ CGIs G Java Servlets

Compiled Element Issues

n Platform dependence

G C/C++ and Java with JNI are platform-dependent G Java without JNI and scripts are independent

G Impacts location of development workspaces

n Deployment location

Supporting Compiled Elements

n C/C++ CGIs

q Compile on server or like platform

q Target or copy to /cgi-bin on virtual host n Java applets

q Compile on any platform (JNI usually non-issue) q Target or copy to virtual host

n EJBs and Servlets

q Compile on any platform q JRE compatibility

q Target or copy to server’s <app-path>/WEB-INF/classes

n java –d <app-path>/WEB-INF/classes Some.class

n Manually update deployment descriptors if necessary

Web Site Deployment

n Perforce can be used to deploy Web sites n WebKeeper not advised for production sites

G Additional DNS lookups and network requests G Potential security hole through firewall

G Large Perforce operations result in Web site

unavailability

Deployment Techniques

n Check in binaries

n Maintain two document trees

G Ensures atomicity of deployment

G Reduces timing demands for deployment G Method

n One is live

n Deploy to the non-live n Switch

Web Site Double-Buffering

Apache

Copy 1 Copy 2

Web Server Perforce Server

p4d

Depot

httpd.conf

Advanced Deployment

n Coordinating deployment to and switchover in

server farm

n Deploying to multiple hosting centers n Sub-dividing content space for partial

deployment

n See P4UC00 presentation by David Markley

and Scott Money of Lycos, Inc. for ideas on large-scale deployment

Branching for Web Content

n Different from branching for traditional

software products

G Shorter cycles

G Smaller functional change sets G More evolutionary

G More continuous

n Typically two- or three-tiered approach

G Development mainline G QA codeline

WCM Branching

Development Mainline QA Codeline

Release Codeline

•Central for ongoing enhancements and fixes •Project branches originate here

•Prevents “code freeze” delays •Allows selective integration •Allows direct fixes of QA defects •Possible simple release codeline

•Prevents QA stall waiting for release •Encapsulates final release configuration •Allows selective integration of

independent components

•Allows final release transformations •Deployment master

For More Info

n Additional info in paper on

G Installing WebKeeper

G Whether and when to archive binaries

G Handling shared resources like databases G Managing Apache configurations

Future Directions

n Expand application server integration

G J2EE servers

G Zope

n Expand large-scale deployment strategies n Enhance WebKeeper

G See

http://public.perforce.com/public/perforce/webkeeper/index.html

G Started basic authentication before conference based on Chris

Seiwald’s work

G Mass virtual hosting support

G Add binary modules to depot

G Support Apache 2.0

Conclusion

n Perforce + Open-Source Software + Best

Practices = Effective Web Development Management

n As always, consider the needs when

implementing solutions

n Hopefully, this material gives you the