Now I’m not sure whether his guidance on decision making process is generic, or whether that process depends on your organisational position and role in determining “the right thing”.  I don’t know whether the key difference in our perspectives is between leadership as a manager versus leadership as an influencer, or the difference between managerial and technical leadership, or the difference between synthesising solutions and deciding which to adopt, but Mike’s article portrays a very different perspective to mine.

Mike portrays as key the ability to focus on key issues, and exclude those which are “not relevant”.

In my experience as an architect and technical leader, I spend a lot of time understanding and analysing the different forces on a problem. These design forces may be technical, or human: financial, commercial or political. The challenge is to find a solution which best balances all the design forces, which if possible satisfies the requirements of all stakeholders. It is usually wrong and ultimately counter-productive to simply ignore some of the stakeholders or requirements as “less important” – any stakeholder (and by stakeholders I mean all those involved, not just senior managers) can derail a project if not happy.

Where design forces are either aligned or orthogonal, there is usually a “sweet spot” which strikes an acceptable balance. The problem effectively becomes one of performing a multi-dimensional linear analysis, and then articulating the solution.

However, sometimes the forces act in direct opposition. A good example, currently personally relevant, is system security, where requirements for broad, easy access directly conflict with those for high security. In these cases the architect has to invest heavily in his skills in diplomacy – to invest a lot of time understanding stakeholder positions. One common problem is “requirements” expressed as solutions, which usually hide an underlying concern which can be met many ways, once understood.

In cases of diametrically opposed requirements, there are usually three options:

1. Compromise – find an intermediate position acceptable to both. This may work, but it may be unacceptable to both, or it may fatally compromise the architecture.
2. Allow one requirement to dominate. This has to be a senior level business decision. As an architect, you then have to be sensitive to whether the outcome is genuinely accepted and viable, or whether suppressing the other requirements will cause the solution to fail.
3. Reformulate the problem to remove or reduce the conflict. In the security example the architect may come up with a cunning partitioning of the system which allows access to different elements under different security rules.

Of course, you can’t resolve all the problems at once – that way lies madness. An architect uses techniques like layered or modular structures, and multiple views of the architecture to “separate concerns”. These are powerful tools to manage the problem’s complexity.

It’s also important to remember that the architecture, and its resolution of the various design forces (i.e. how it meets various stakeholder needs) have to be communicated to many who are not technical experts. The technical leader must take much of this responsibility. I have had great success with single-topic briefing papers, which describe aspects like security in business terms, and which are short and focused enough to encourage the readers to also consider their concerns separately.

One area where I do agree with Mike is the need to listen to the voice inside, and carry decisions through with integrity. For an architect, the question is whether the architecture is elegant, and will deliver an adequately efficient, reliable and flexible solution. If your internal answer to this is not an honest “yes”, you need to understand why not, and decide whether you and your users can live with the compromises.

And finally, the architect must protect the integrity of the solution against the slings and arrows of outrageous projects. Monitor in particular those design aspects which reflect compromises between design forces, because they will inevitably come under renewed pressure over time. You have to not only do the right thing, but ensure it is done right.

Non-Sequiteur

About the weird title: Mike is attempting to create his blog based largely on 1970s Prog Rock references. As a tribute to such an excellent idea, I feel compelled to join in (at least on this occasion)!

I recently converted my blog from an old bespoke (= “custom”, for my American friends) solution to one based on WordPress. However, this created a problem, in that the WordPress model is to hold all content in the database, and that wasn’t the right model for me.

Firstly, I have a number of articles which are very long for a blog post, and I had no interest in restructuring them. I also didn’t want to break external links to the existing articles.

Next, I decided that I wanted the freedom to continue to write in that style. Some of my writing takes several weeks, and it works for me to draft it as separate HTML pages. I also sometimes want to include active content or multiple images, and I don’t want to create a large and unwieldy WordPress database full of such stuff.

Finally, my online photo galleries are managed and generated using Jalbum, and I wanted to find a way of neatly integrating single images into my blog, complete with the watermarks and metadata extraction which Jalbum manages so well, without duplicating that functionality in WordPress.

This is probably typical of many older web sites, but WordPress doesn’t really embrace the integration of external content. This article describes how I solved this problem, and a WordPress plugin I have developed to make my solution reusable.

I have therefore now created a number of new “views” of my blog, and a number of specialist feeds, as follows:

• Thoughts on the World. This is the original and complete blog and feed, and will give you access to all my content, in chronological order. Expect a mix of articles including my observations of the world of software development and architecture, humorous observations, photography and personal announcements.
• The blog address is www.andrewj.com/blog
• The feed is http://feeds.feedburner.com/ThoughtsFromAndrewJohnston
• The Professional Blog. This will focus on professional articles based around my observations of the world of software development and architecture. Articles for AgileArchitect.org will appear here, as will updates on my products and publications, and I may cross-post relevant reviews. You’ll also see any practical announcements relating to my websites and feeds.
• The blog address is www.andrewj.com/blog/pro
• The feed is http://feeds.feedburner.com/StructureOfTheWorld
• The Photo Blog. This will focus on my photos, and photography-related articles and reviews. You’ll also see any practical announcements relating to my websites and feeds. There’s also a separate feed for updates to my gallery.
• The blog address is www.andrewj.com/blog/photo
• The feed is http://feeds.feedburner.com/ImagesOfTheWorld
• The feed for my gallery is http://feeds.feedburner.com/GalleryOfTheWorld
• The Review Pages. This presents my reviews of books, films and music. You’ll also see any practical announcements relating to my websites and feeds.
• The blog address is www.andrewj.com/blog/reviews
• The feed is http://feeds.feedburner.com/ReviewsOfTheWorld

For more details, visit my Blog Views and Feeds page.

My new blog is now live. This allows a lot more flexibility, including commenting etc.

Please can you make sure you are taking your RSS feed from one of the following:

http://feeds.feedburner.com/ThoughtsFromAndrewJohnston (preferred)

http://www.andrewj.com/blog/index.php/feed/ (straight from my site)

I still need to update some of the links to longer old articles, so please bear with me if these are a bit odd over the next few days. Also you may see some duplicate items in the Feedburner feed – if so, please ignore these, and they should clear shortly.

Thanks, and enjoy the new blog.

Andrew

Migration of our web sites and email to the new hosting server is almost
complete. See previous article/rant for some of my findings. After some initial
frustration with my hosting provider (WebFusion) when I discovered the
limitations of their new Linux shared hosting services compared with the old
one, I have to say a big “Thank You” for their efficiency in finding a better
solution for me, and providing me with effective technical support to get it up
and running.

If you do have any problems with our sites or email, let me know…

I’m currently developing a new blog, based on WordPress. This will make it
easier to post “on the fly” than with the current solution, hand-carved from XML
and ASP (now PHP).

When the new blog is running, I’m going to have a regular post for fans of my
photography, so you can see what I’ve been working on. In the meantime, I’ve
updated my gallery pages so they are a bit easier to navigate, bookmark and
search.

And talking about my photography, I’ve recently been accepted by the Alamy
stock agency. So please all rush at once and spend lots of money licensing my
pictures for all those uses you’ve dreamed of but were too polite to mention

See you soon,

Andrew

I’ve just spent a week-end migrating my website to a new hosting server. As
part of that process, I had to rewrite all my old ASP code using PHP. Here’s
what I learned:

1. The Apache/Linux community have misleadingly changed the meaning of “ASP”.
   If you bought a Linux-based hosting service 5+ years ago with “ASP”, it meant
   a *nix port of Active Server Pages. That worked for me, as I could develop it
   on Windows. Now, if you buy a Linux hosting service with “ASP” it means
   “Apache Server Pages”, and the embedded language is Perl. Useless!
2. PHP has positively the worst combination of features for a language:
   • A c-based language’s sensitivity to case, ending semicolons and curly
     bracket counts,
   • None of the protections against errors in the latter that a C++/Java (or
     VB) language gives you, like strong typing and forced variable
     declaration,
   • No single-step debugging. Now I accept that this may not be 100% true,
     so don’t all write in with the names of all the debuggers I didn’t find in a
     quick search for tools on Sunday morning, but certainly I don’t have one at
     the moment,
   • It runs differently on Windows and Linux, and in a way I haven’t yet
     understood 100%, so I can only test by uploading to my live
     website.

That said, I’ve still got it! I’ve managed to convert my blog and my book
reviews, and I’ve actually improved on my old code for the latter. Just please
let me have VB.NET back for my next major project.

OK. </rant>

Overall there was much less focus on technology (read SOA and modelling tools) this year, and a vigorous and renewed focus on business alignment and business architecture, which, if we can deliver, potentially places architecture where it should be, as the business’s agent.

But there’s a problem. Good business analysis is fundamental to this, yet several delegates bemoaned the current lack of good business analysts. User organisations often struggle to articulate and abstract their needs, and this feeds into all downstream processes. Modelled requirements are an increasing rarity, poorly substituted by imprecise verbal statements in Word or PowerPoint.

The problem is, of course, not unique to analysts, and may have common cause with the equal lack of architects. Senior architects and analysts both tend to have several big birthdays under the belt, and many learned their trade as developers, gaining both practical method skills and the experience of turning ideas into working code. (The majority of exceptions have other “making it work” experience, such as building networks or running data centres.)

But in the current world of ERP packages and large-scale outsourcing, many organisations no longer build anything themselves. The live classroom has been thrown away.

I have worked with a number of good, keen young analysts, but most work for large supplier companies who still have both well-funded training programmes and the breadth of work to build experience and a broad skill set. These guys and girls can do a good job, but at the risk of higher costs and potential conflicts of interest.

We already know that this may reduce organisations’ ability to ensure the right solution to their needs, or assure its quality. Recent observations suggest that organisations who forgoe getting their hands dirty in IT will also suffer an increasing difficulty in creating a clear, concise and structured statement of those needs themselves.

I explore two very different examples: the value of understanding RS232 in this 3G+ world, and some recent discussions on service
reliability, both of which can be resolved using some quite old
knowledge…. (Read More…)

I presented a paper on Agile Architecture. If you regularly read my work you’ll recognise many of the ideas, but I’ve managed to bring them all together for the first time. You can download my slides and script here.

What was very interesting was how the thrust of the material has changed from a few years ago. No-one was claiming that a given framework, process or toolset can solve EA problems. At the risk of being uncharitable I thought John Zachman’s ideas sounded very tired, and there was almost no mention of alternative frameworks such as TOGAF. I may have self-selected by not attending any vendor sessions, but there was also no promotion of tools or technology. A common view was that EA, SOA and many supporting concepts are currently entering the trough of the hype cycle.

Instead the focus was largely on people-related problems and approaches. The labels varied, but several speakers introduced ideas familiar to agile architects. Maybe we’re doing something right after all.

Why you need architects, in song and dance.

Enjoy!

As a first step, I am trying to develop a comprehensive list of test automation “best practices”, grouped roughly by life-cycle (or iteration) stage. I’m trying to find practices which are broadly independent of specific methods and technologies, although obviously tool support may vary depending on the chosen technology.

This article is my first draft of such a list.

I’d welcome suggestions from my readers if you think there are any omissions (or if you substantially disagree with anything I’ve included).

Thanks

Andrew

I’m presenting a paper entitled “The Agile Architect”. This focuses on both how agile projects can have a strong architecture, and how architects can learn and benefit from agile approaches. I take a rather different approach to some recent papers with a similar title (e.g. at this week’s otherwise excellent Microsoft Architecture Insight conference), which suggest that agile projects can “do away with the architect”.

I look forwards to seeing you there.

Andrew

But I’m not sure that’s the key question. I think the right question is “Who are the architects?”

Somehow, a number of tasks must be discharged, but how varies from project to project. In the last year I’ve had a modest building project which tells an interesting story about how different people contribute to “the architecture”. Read more here…

This is a story of a project I worked upon which followed an informal, agile process, but delivered a successful architecture. Hopefully it serves to support my assertion that agile can have an architecture, but needs an agile architect.

However, UML doesn’t help when it comes to the transformations themselves, and typically the detail has to be captured either in code, or a proprietary format. Most good integration tools provide some sort of “visual mapping tool”, where the developer drags and drops to create links between representations of the two structures, usually imported directly from their physical schemas. Here’s an example using SeeBeyond. Altova provide a good stand-alone data mapping tool called Mapforce – here’s an example showing it in use. The problem is that these tools work directly with the physical structure, and don’t export the mapping information in a reusable format, so that information is completely disconnected from the UML analysis or design models.

I have experimented with trying to represent mapping information in a UML model, but so far without much success. The best solution I’ve found so far is to use some sort of “pseudo code” (it could be OCL, pseudo-Java, pseudo-VB or anything similar). For example, we could easily annotate the model with code fragments such as:

Database.order_table.order_no = Message.Header.OrderNo

(where each element refers to a UML Package.Class.Attribute combination).

The problems are that it’s not clear where to put this annotation, most UML modelling tools won’t help generate it, and there’s no graphical representation. Ultimately, writing pseudo-code like this is probably not much better than abandoning the model and moving straight to using your integration tool.

My question is: does anybody know a better way? Has anybody found a good way of representing mapping information in UML? And if so, is there any good tool support?

If you know, please send me a message.

While the metaphor might still be valid, some people are beginning to question how far it should be taken. Helland’s article, like others before it, implies that “good” EA looks rather like a medium-sized modern American town, complete with relatively standard services, civic buildings and commercial venues. In an answer to the original “Metropolis” article Richard Veryard and Philip Boxer have published “Metropolis and SOA Governance” which challenges several of Helland’s assumptions.

I think that maybe we should extend the metaphor by thinking about cities, or Enterprise Architectures, as very diverse entities. What sort of “city” do you live in? To what extent is it planned? What is the vision, and do the citizens share in it? Does the EA resemble a nice neat midwest town, a dark, brooding Gotham City, a glass and steel Utopia, a federation of small towns with lots of empty space between them, a medieaval walled town, or a wartime mid-european ghetto?

And the metaphor can be taken further. Do you want to promote “infill development”, closing up functional gaps, or do you want to keep clear separation between the various zones? Do you want the shared services to be clearly visible, as they are in modern, purpose-built towns or hidden beneath a facade which looks much older or simpler? Do you expect to eventually knock down and rebuild older “legacy” zones, or do you want to preserve them with the minimum of change (a common requirement for our valuable historic buildings)? Do you want to accomodate the small hardware shop (read small the bespoke system) as well as the new DIY superstore (the ERP package)?

Finally, remember that it is extremely rare for a city to be truly planned and designed from scratch. You usually start with something established. Even if the city has been flattened by a bomb, you’ll have to observe land rights (this is what stopped Christopher Wren and Charles II realising their grand design after the Fire of London). This is equally true of Enterprise Architectures.

The city planning metaphor is a powerful one, but its true power may come if we use it to explore problems as well as utopian ideals.

Keep reading!

Andrew

Three short quotes from Helland’s conclusions bear repeating directly. On heterogeneity he says:   

Remember that heterogeneity happens. Unless you have a very simple application portfolio, shared services will not be achieved by trying to put all
of your applications on one version of one platform. Even if you could, the next
merger would change that! Rather, you have to design for interoperability and
integration across platforms. This is the force that is driving the industry
wide work in service-oriented architectures.

He extends the popular “city planning” metaphor to IT investment:   

IT investment is a balance of funding the sacred, protecting historic monuments, and allocating spending between infrastructure and business opportunity. Striking this balance is a key facet in effective governance, and in realizing the potential of IT in your organization.

And finally, those who seek to maintain control of their enterprise
architecture through heavy governance would be well advised to note: 

You have to maintain a light hand. It is counterproductive to try to dictate
what happens in every structure in town, what color shirts are made, and how much is charged for soap. You have to embrace the semi-autonomous approach to governance that is characteristic of our cities, and allow the process owners to optimize and achieve efficiencies with as few constraints as
possible.

However, Martin’s essay goes beyond common programming and data techniques to look at the development of specialist tools which he calls “Language Workbenches”. I’m not completely convinced that we need these in the world of XML and XSD. If you have a defined schema for you XML-based DSL (and aren’t all the many *ML langauges just different DLSs?) then any schema-sensitive editor will provide you with good design and editing support. The leading IDEs (e.g. Visual Studio) all have such a tool built into their core capabilities. Surely we now have a sufficiently sophisticated set of XML-based tools and standards that we have an opportunity to exploit synergies rather than re-inventing the wheel?

In a paper entitled “The New Methodology” Martin Fowler, the guru of object-oriented development, suggests a couple of reasons why this might be.

My article answers Martin’s, suggesting a couple of other
considerations, and whether we have to completely abandon the physical construction analogy as a result.

Almost all of the presentations shared a reminder that we still have a “software crisis” – the vast majority of software projects fail to deliver to their original targets and estimates. The presentations suggested three independent, but not exclusive, approaches to try and resolve the problem:

• Adopting better, more agile processes to address fundamental weaknesses in “waterfall” processes,
• Adopting better tools and techniques to improve development productivity and the integration of the application life-cycle,
• Enforcing a stronger “enterprise architecture” framework for development.

This last one was surprising, with several papers echoing my view that a strong architecture is essential if agile development is to succeed on a large scale or in complex and critical applications.

There was also surprising agreement on things which won’t solve the problem:

• No-one was promising a technical or product “silver bullet”. This includes web services!
• No-one was suggesting that we should just “try harder” with old-fashioned tools and processes.
• There’s no “one size fits all” solution. For example it’s a mistake to force a formal, high-ceremony process onto small business systems developments.
• Excessive technical standardisation is also not the answer. The drawbacks include “lowest common denominator” technical solutions and inflated costs where the standard solution is “overkill”.

Read my report for more details.

Steele’s argument is highlighted by the paper’s unique style – he uses a small but growing language himself. He writes using only words of one syllable, gradually adding other words he has defined in these terms.

The paper works at many levels. As well as the fascinating intellectual exercise in style, it makes a strong case for:

• simple but extensible programming languages,
• improving the extensibility of Java, rather than its core vocabulary,
• an agile community process for developing languages, rather than up-front design of great monoliths,
• the communication power of simple language and words.

Steele exhorts us to think about extensibility mechanisms – if we get these right then the core functionality can always grow. And by example, he encourages us to use simple, direct language and benefit from its discipline. On both accounts I agree wholeheartedly.

In my recent work I’ve discovered a few patterns of my own, and I’ve started a page to document them.

First up is entitled “Change Indicator” . You may have a legacy system with an EAI adaptor, or a similar source of messages, which reports on the current state of key business objects. The messages will typically tell you when something has changed, but not necessarily what has changed, but the latter may be important to downstream systems. This pattern shows how to use EAI elements to add this information in a way which is totally transparent to the legacy system and existing users of the EAI scheme.

But Gergor’s article doesn’t really touch on understanding the dynamics of business change, and how these should dictate the form of architectural extensibility. Neither does he discuss the problem of being an architect who understands what flexibility is required, but being unable to get either the customers, or the suppliers, to agree. I seem to spend a lot of my time saying “I told you so” when a problem I predicted occurs – this is frustrating, even if I’m being proven right.

If you want to understand more about the dynamics of change, you might look at my previous postings on the subject:

Architects
- Masters of Order and Unorder

Business
Flexibility

Strategies
for Flexibility

One recent experience suggests to me that we may well be at risk of this. Read about my first experience of Web Service development and judge for yourself: have we thrown the baby out with the bath-water?

Of particular interest, there’s an article in the first edition by Cap Gemini on “Enterprise Architecture Design and the Integrated Architecture Framework”. Obviously every consultancy has their own architecture framework, and they all share some
features, but what makes this one a bit different is the strong focus on the contextual (business context), conceptual (vision) and logical (solution independent capability) aspects, with technical details relegated to a bottom physical layer. It also has one of the best discussions I’ve seen on the relationship and differences between enterprise and project architectures.

For the technically inclined, I’ve built my blog using a combination of VB.NET, Active Server Pages, and a bit of XSLT. This allows me to develop my site using FrontPage and IIS, and run it under Apache. If anyone’s interested I’ll write a technical note on it.

This rather tongue in cheek piece was inspired by a “group therapy” session at the IRM/Meta group Enterprise Architecture Conference 2004.

This article, co-written by Richard Wiggins and originally published in the Rational Edge in February 2003 describes a new approach, based on UML, which meets the real requirements of modelling the Enterprise Data Architecture.

However, they rely entirely on Monte-Carlo simulation. I believe that where the curve is defined by, or can be approximated by, a few discrete points, a relatively simple analytical solution can then be used in place of simulation.

This paper (in Adobe Acrobat Format), originally published by the CBDi Forum, presents some of those strategies.