Creating Drupal content using Python
I wanted to import data into Drupal from elsewhere - in my case, from XML exported by another system.
Perhaps the easiest way to interface with Drupal at a simple level is to use the Services module. This lets you, for example, create applications in other languages which can create, list, or update Drupal content. There is a mix of documentation for it, some of which is a bit dated.
So here’s a quick example of how you can use a Python program to create a node in Drupal. Hope this helps to get you started if you need to do something similar.
#! /usr/bin/python
#
# Creating nodes in Drupal 6 via the Services module using
# Python and XML-RPC.
#
# To use this, get and install the Services module from
# http://drupal.org/project/services
#
# Enable the XMLRPC Server module, the System Service module and the
# Node Service module at least.
#
#
# Note that this is intended to demo quick importing of data from other
# systems. It makes no use of authentication, in fact you need to enable
# various permissions for anonymous users to be able to use it.
# Enable things like 'administer nodes', 'create page content',
# 'create url aliases' and 'administer url aliases'.
#
# You should remember, therefore, to DISABLE them before you go live
# with a production site!
#
# This assumes you have 'Use sessid' but not 'Use keys' enabled in the
# Services settings at http://yoursite/admin/build/services/settings.
import xmlrpclib, time
# Put the URL for your Service in here. See admin/build/services.
s = xmlrpclib.ServerProxy('http://myhost.com/services/xmlrpc')
class node:
# You need to set uid and username appropriately for your site if you don't want
# everything to be posted by Anonymous.
def __init__(self, title, body, path, ntype='page', uid=1, username='qsf'):
self.title = title
self.body = body
self.path = path
self.type = ntype
self.uid = uid
self.name = username
self.promote = False
try:
sessid, user = s.system.connect()
# Here you could read in the content for each node from some other source and do
# n = node(title, body, path)
# but for now we'll just do:
n = node('A test node', 'This is an interesting page', 'interesting')
# and then save it into Drupal
s.node.save(sessid, n)
# where it should appear at /interesting.
except xmlrpclib.Fault, err:
print "A fault occurred"
print "Fault code: %d" % err.faultCode
print "Fault string: %s" % err.faultString
Importing content from Plone to Drupal
I used a variant of this script to convert a Plone-based site to Drupal. Whether any of the following would work for somebody else’s site depends to a great degree on the particular configurations involved, but I offer it in case it’s useful as inspiration! You’ll almost certainly need to tweak it.
As those familiar with Plone will know, the content is stored in an object-oriented database, and mapping that onto other things can be tricky. Most of my content was reasonably straightforward, though, and I tackled the problem by switching on the Zope options to make the Plone site accessible by FTP. I could then use the FTP tool of my choice to access what looked like a filesystem and copy the whole hierarchy onto another machine.
My Plone content was basically HTML and images, and each node in Plone
came across as a simple HTML file without all the boilerplate
formatting. Perfect! I could then write a script which would take, for
example, the file in news/article1, read it as XML, gather the bits I
wanted and create a node in Drupal which also had news/article1 as the
path.
The main bit of manual tweaking was to do with relative URLs. In Drupal
you really want every link on your site to be an absolute URL (ie. to
begin with ‘/’), because there are so many ways of chopping and
displaying your content. On the old site, news/article1 might link to
article2 in the same folder, which isn’t going to work if
news/article1 is sometimes accessed as node/29, or
recentstuff/thisweek, or whatever. You need to look for links that
don’t begin either with http or with / and update them
appropriately to /news/article2 or whatever. I didn’t do that in the
script because there were few enough that I could do it using global
searches in my editor.
OK, so here’s my import.py script. Having copied the Plone content to my local drive by FTP, I changed into the folder and did, for example:
./import.py news/article1 news/article2 ...
OK - here it is. I hope it’s reasonably self-explanatory. I use the tidy utility to convert sometimes non-conformant HTML into valid XML, and
#! /usr/bin/python
#
# This assumes you have 'Use sessid' but not 'Use keys' enabled in the
# Services settings at http://yoursite/admin/build/services/settings.
# You'll need the appropriate permissions set, and the Path module enabled.
import xmlrpclib, time, sys, subprocess, os
import xml.etree.ElementTree as ET
# Where is the XML-RPC service on your Drupal site?
s = xmlrpclib.ServerProxy(':8888/services/xmlrpc')
class node:
# You need to set uid and username appropriately for your site if you don't want
# everything to be posted by Anonymous.
def __init__(self, title, body, path, ntype='page', date=None, uid=1, username='qsf'):
self.title = title
self.body = body
self.path = path
self.type = ntype
self.uid = uid
self.name = username
self.promote = False
self.format = 3
self.comment = 0
if date:
# self.created = date
# self.changed = date
self.date = date
print "date = ",date
sessid, user = s.system.connect()
for path in sys.argv[1:]:
try:
# Read the XML file, tidying it up and making valid XML.
tidypipe = subprocess.Popen(["tidy", "-q", "-asxml", "-n", path],
stdout=subprocess.PIPE)
# The 'tidy' process will sometimes add namespace declarations.
# ElementTree stuff below gets messy if we have to parse namespaces
# so I'm just going to throw them away before we read the file.
nsstrip = subprocess.Popen(["sed", "s/^<html .*>$/<html>/"],
stdin = tidypipe.stdout, stdout=subprocess.PIPE)
# Read the file into a DOM tree
tree = ET.parse(nsstrip.stdout)
root = tree.getroot()
desc = None
date = None
ntype = 'page'
# Some of the info we want is in the <meta> tags.
for i in root.findall(".//meta"):
if i.get('name')=='Description': desc = i.get('content')
if i.get('name')=='Effective_date':
date = i.get('content')
if i.get('name')=='Type' and i.get('content')=='News Item':
ntype='story'
# This is where we build up the body of our new node.
body = ""
# We turn what was the 'Description' in Plone into the teaser text in Drupal
if desc:
body += "<p>%s</b>\n<!--break-->\n" % desc
# The title comes from <title>
title = root.find('.//title').text
# and the rest from <body>
body_outer = root.find('.//body')
# The <body> may have text that isn't inside any element.
body += body_outer.text
# then we add all the elements.
for n in body_outer:
body += ET.tostring(n)
# Strip off index_html, if that's part of the path
path = path.replace('/index_html', '')
# Show the user what's happening
print "Title:",title
print "Desc: ",desc
print "Date: ",date
# And create a new node
n = node(title, body, path, ntype=ntype, date=date)
s.node.save(sessid, n)
# On a Mac this will open it in the browser so you can check it out.
# os.system('open %s/%s' % (':8888', path))
print "------------------------------------------"
except xmlrpclib.Fault, err:
print "======================================"
print "A fault occurred with",path
print "Fault code: %d" % err.faultCode
print "Fault string: %s" % err.faultString
print "======================================"
I don’t normally have comments switched on on this site, so if you’d like to add any comments/tips, it might be best to do so on the Status-Q post.
Quentin Stafford-Fraser, Dec 2008.
