Thanks to Rory for thinking up the name. :-)
Shipd is a way of abstracting the shipping information from minivend and running it from a standalone server.
Last update Aug 24th, 2000.
Aug 24th, 2000 : Now I'm going to try and write a simple module that will use the new shipd.conf file to provide simple 'box orientated' pricing. That is after I've finished the new Shipd_config.pm. I expect that foreach diferent way of defining shipping costs there will be a module. Each module will be disscribed by a tag in the shipd.conf file.
Aug 24th, 2000 : After finishing the module for shipd to provide it with a configuration file I've scrapped the shipd.conf format and rewritten it in what I hope will be better style for people( and myself) to impliment, hey, ho back to the drawing board for Shipd_config.pm. See version 0.0.6 here.
Jul 17, 2000 : Updated the Shipd_config.pm module to read version 0.0.5 config files. See the output.
Jul 17, 2000 : Finish a modules that reads in the configuration file ( version 0.0.3) and returns it to the modules caller as a ref to a nested hash statement. It looks pretty under Data::Dumper. For those who are interested it is here, Shipd_config.pm.bz2, it is compatable with version 0.0.4 config files it just won't use the new tags.
The next stage wil be upgrading it for version 0.0.4 config files. Then comes the task of taking the hash ref and turning it into shipping code :-). Oncethat is done I will roll out a new version of the server.
Jul 13, 2000 : Created a provisional configuration file for shipd. This will be interpreted by shipd providing generic solutions for calculating shipping costs. For example calculating shipping cost by weight or quanitiy of a mixture of both. Defineing maximum parcel size etc. Look here, please comment!
Jul 12, 2000 : An rc.d script for shipd can now be found here, shipd.gz. Make sure the server is started before minivend !
Normally minivend uses the mv_shipmode to calculate the shipping cost of an item. This works fine for single shops but when creating malls there becomes a problem of scalablility. What I wanted to do was have a way for multiple shops to access each others shipping data without filling each shipping.asc file in each shop with the data from all the other shops. Effectivly the shipping.asc file would grow very large almost proportional to number_of_shops squared.
To do this I stopped using mv_shipmode and left it undefined. This efectivly sets it to default. Then I used mv_ship_mode as the variable for shipping and salex tax.
This is the variable that is now passed to the server that lets it where the destination of the package is.
When the basket page is displayed a loop is rolled over all the carts that have been shopped in, ie items A & B from cart shop_1 and items C and D from shop_2. Every time the loop is itterated the value of a scratch variable present_cart is set to the name of the cart/cataloge. This is passed to the server, letting it deside what courier to use for that shop. Two more subroutines in minivend.cfg package up the information from the cart, like weight and quantity of each item, and pass it onto the server. The server uses the information to return a single figure, the total cost of shipping for that cart.
All that is left is to gather up some imformatiuon about the product itself and pass that into the server. To do htat I wrote a global subroutine gen_hash
GlobalSub <<EOGA
sub gen_hash {
use strict ;
# this subroutine generates a hash of quanitity and
# wieghts for the shipping routines.
# This is mainly for backwards compatability.
# This is the cart we are going to look over.
my $cart = shift ;
my $Carts_ref = shift || [] ;
my $Tag = shift ;
# So $Carts->{$cart} contains all the info we need on the quanitities
# if each product.
# '$Carts->{$cart}' points to an anonymous array. Each element
# ( read item) in the array is a hash ( one hash per item).
# The value to the key 'quantity' tells us how many
# of each item is being bought. The vakue to the key
# 'code' gives us the unique code for the product
# for that basket.
# to get the weight we use the construct
# '$Tag->data($table, $field, $key)'
# So '$Tag->data("product","weight",)'
# should return us the weight of the product.
# We start by creating the hash we will return.
my %data;
# Then loop over each item in the cart.
foreach my $item (@$Carts_ref){
# each item is contain in an
# anonymous hash, pointed
# to by the ref '$item'
# This items code is
# $item->{'code'}
my $code = $item->{'code'} || "B40";
# and simular for the quanitity
my $quant = $item->{'quantity'};
# now lets get the weight
my $weight = $Tag->data('some_shop','weight','some_product-code');
if (! defined $weight){
$weight = "un";
}
# with the three parts of the puzzle we can
# put them into '%data'
# first another anymous hash
my $hash = {
code =>$code,
quantity =>$quant,
weight =>$weight
};
# the add it to the '%data'
$data{$item} = $hash;
# even thougth we are usig a hash as a key
# that is alright. We don't loos any data
# but does alloes us to have unique keys !
}
# right thats it lets return are structure
return \%data;
# return $weight;
# return 1;
}
EOGA
Next was a routine that would send it to the server and return the cost.
GlobalSub <<EOGS
sub send_message {
#use strict ;
use Socket ;
use Data::Dumper;
my $data = shift ;
my $d = Data::Dumper->new([$data]) ;
$d->Purity(1);
$d->Indent(0);
my $data = $d->Dump;
my ($remote,$port, $iaddr, $paddr, $proto, $line);
$remote = 'localhost';
$port = 3000; # random port
if ($port =~ /\D/) { $port = getservbyname($port, 'tcp') }
die "No port" unless $port;
$iaddr = inet_aton($remote) || die "no host: $remote";
$paddr = sockaddr_in($port, $iaddr);
$proto = getprotobyname('tcp');
socket(SOCK, PF_INET, SOCK_STREAM, $proto) || die "socket: $!";
connect(SOCK, $paddr) || die "connect: $!";
my $old = select(SOCK);
$| =1 ;
print SOCK "$data\n";
my $responce ;
chomp ($responce = <SOCK>);
$| =0;
select($old);
close (SOCK) || die "close: $!";
undef $d;
return $responce;
}
EOGS
With all this the shipping.asc file for each and every shop becomes...
code description criteria min max formula
default United Kingdom quantity 0 99999999 f [perl subs=1 global=1 ]my $null = gen_hash($Scratch->{present_cart},$Carts->{$Scratch->{present_cart}},$Tag); my $cost = send_message([$null,$Scratch->{present_cart},$Values->{mv_ship_mode}]);return $cost;[/perl]
Where before I used minivends internal mv_shipmiode I now use mv_ship_mode, i.e. in basketcontent.html a part of basket.html
<b>Shipping Destination:</b>
<SELECT NAME="mv_shipmode">
__SHIPPING__
</SELECT>
<b>Shipping Destination:</b>
<SELECT NAME="mv_ship_mode">
__SHIPPING__
</SELECT>
In my catalog.cfg file I also made some changes ( note my catalog.cfg file are hand written not generated, this will change soon , so you may notice some diferences from the catalog.cfg files that come with e example shopd in minivend).
ValuesDefault mv_ship_mode SCO
...
CustomShipping code
...
NoImport shipping
...
Database shipping shipping.asc TAB
## Default: default
#
# Sets the initial shipping mode.
#
#DefaultShipping GBR
...
SalesTax mv_ship_mode
...
## Default: 0
#
# A flat shipping charge that will be added to any order. A
# zero value disables it.
#
# Shipping
...
Variable SHIPPING <<_EOF_;
[sql type='list' query='select code,description from shipping']
<option value="[sql-code]" [selected mv_ship_mode [sql-code]]>[sql-param description]</option>
[/sql]
_EOF_
All thats left is to create the server
#!/usr/bin/perl -w
use strict;
BEGIN { $ENV{PATH} = '/usr/ucb:/bin' }
use Socket;
use Carp;
use Sys::Syslog qw(:DEFAULT setlogsock);
use POSIX;
use DBI ;
# make our selves into a daemon
my $pid_master = fork;
exit if $pid_master;
die "Couldn't fork : $!" unless defined($pid_master);
#sleep(5);
open(PID,">/var/run/shipd/shipd.pid") || die "$!";
print PID $$."\n";
close PID;
POSIX::setsid() or die "Can't start a new session: $!";
my $EOL = "\015\012";
sub logmsg {
setlogsock("unix");
openlog("shipd","nodelay","user");
syslog("info","$0 $$: @_");
closelog();
#print "$0 $$: @_ at ", scalar localtime, "\n"
};
my $port = shift || 3000;
my $proto = getprotobyname('tcp');
$port = $1 if $port =~ /(\d+)/; # untaint port number
socket(Server, PF_INET, SOCK_STREAM, $proto) || die "socket: $!";
setsockopt(Server, SOL_SOCKET, SO_REUSEADDR,
pack("l", 1)) || die "setsockopt: $!";
bind(Server, sockaddr_in($port, INADDR_ANY)) || die "bind: $!" ;
listen(Server,SOMAXCONN) || die "listen: $!";
logmsg "server started on port $port";
my $waitedpid = 0;
my $paddr;
sub REAPER {
$waitedpid = wait;
$SIG{CHLD} = \&REAPER;
}
$SIG{CHLD} = \&REAPER;
for ( ; $paddr = accept(Client,Server); close Client) {
my($port,$iaddr) = sockaddr_in($paddr);
my $name = gethostbyaddr($iaddr,AF_INET);
logmsg "connection from $name [",inet_ntoa($iaddr), "] at port $port";
request_cost();
}
sub request_cost{
my (@lines,$line,$data);
# do a fork
my $pid ;
if(!defined($pid = fork)){
logmsg "cannot fork: $!";
return;
}
elsif($pid){
return; # i'm a parent
}
chomp($line = <Client>);
# now we have each line of the dat structure
# lets us re-assemble it.
my $data_ref = eval $line;
# great now we have our structure back.
my (%hash,$cart,$dest,$totalcost) ;
if (ref($data_ref) eq "ARRAY"){
#print "we have an array\n";
my @array = @$data_ref;
if (ref($array[0]) eq "HASH"){
#print "Found HASH ref\n";
# ok so lets assume this hash
# contains the qunatity and
# weight info.
%hash = %{$array[0]};
}
$cart = $array[1];
#print "Cart $cart\n";
$dest = $array[2];
#print "Dest $dest\n";
}
logmsg("Cart : $cart");
if($cart eq "some_shop"){
logmsg("Doing some_shop");
$totalcost = some_shop(\%hash,$dest);
logmsg("Cam Totalcost: $totalcost");
}
else{
$totalcost = 0;
}
my $old = select(Client);
$| =1 ;
print Client $totalcost,"\015\012";
$| =0;
select($old);
exit;
}
sub some_shop{
my ($hash,$shipdest)=@_;
my $numbers;
my $totalweight;
my @weights;
my @zones = get_zones("Royal Mail",$shipdest);
my $z = join(":",@zones);
logmsg $z ;
logmsg "$shipdest";
my $cost = "3.99" ; #set default
$cost = "3.99" if(grep{/wz1/} @zones) ;
$cost = "3.99" if(grep{/europe/} @zones) ;
$cost = "3.99" if(grep{/eu/} @zones) ;
$cost = "1.99" if(grep{/uk/} @zones) ;
# log onto database
my $dbh_some_shop = DBI->connect('DBI:Pg:dbname=minivend') or warn "Coul
d not connect to shipping_def\n";
foreach my $item (keys(%{$hash})) {
my $code=($hash->{$item}->{'code'});
my $statement = "select shipping from some_shop where code = ? ";
my $sth = $dbh_some_shop->prepare($statement);
$sth->execute($code) ;
my $ary = $sth->fetchrow_array ;
if( (defined $ary)&&($ary) ){
logmsg("found book");
$numbers++;
}
$sth->finish ;
}
$dbh_some_shop->disconnect ;
logmsg("Cost : $cost");
if(! $numbers){
return 0;
}
else{
return $cost;
}
}
sub get_zones{
my $biz = shift ;
my $country_code = shift ;
chomp $country_code;
logmsg "So far ok $biz $country_code";
if($country_code !~ /[A-Z][A-Z][A-Z]$/o){
die "Error in country code $country_code\n";
}
my $dbh_zone = DBI->connect('DBI:Pg:dbname=minivend') or warn "Could not con
nect to shipping_def\n";
my $found_table = grep {/shipping_zones/} $dbh_zone->tables ;
if( ! $found_table){
logmsg "Can't find table shipping_zones \n";
}
#else{print "Access to shipping_zones\n"}
my $statement = "select zones from shipping_zones where business = ? ";
my $sth = $dbh_zone->prepare($statement);
$sth->execute($biz) ;#or die "Wibble 2 \n";
my $ary = $sth->fetchrow_array ;
my @ary = split(/\s+/,$ary);
#print join(":",@ary),"\n";
#foreach my $zone (@ary){
#print "$zone\n";
#}
my @zones ;
foreach my $zone (@ary){
$statement = "select $zone from shipping_def where $zone = '1' and code
= ? ";
$sth = $dbh_zone->prepare($statement);
$sth->execute($country_code) ;#or die "Wibble 2 \n";
my $ary = $sth->fetchrow_array ;
next if (! defined $ary);
next if (! $ary);
my @ary = split(/\s+/,$ary);
print $zone,"\t",join(":",@ary),"\n";
push @zones,$zone ;
select(undef,undef,undef,0.0001);
}
$sth->finish ;
$dbh_zone->disconnect;
return @zones ;
}
| business | zones |
| Royal Mail | uk europe eu wz1 wz2 |
The secound table lists all the distinct destinations in the first colunm as three letter codes for the destinations themselves. The secound colunm contains the actual names of the countries. The fifth colunm is a boolean for whether the destination is a country or not. The fourth colunm is a bool if they are inside the UK or not ( this could easily just be USSR or USA etc). The rest of the feilds corespond to the shipping zones defined in the database above. One per colunm. These are also bools.
| code | country | uk | non-country | europe | eu | ... |
| SCO | SCOTLAND | 1 | 0 | 1 | 1 | ... |
I then put these tables into Postgresql, however once I get the config file included into shipd the info will be able to be put into several places pointed to by the configuration.
This is a provisional format for the shipd.conf file. This should remove some of the pain from creating custom shipping solutions. Up to now each different shop would have to be hard coded into the server. Now with simple file, in the style of apache's httpd.conf virtual host syntax, I hope to create solutions on the fly. My efforts on shipd are going to be focused in this direction. When the server has correctly been programmed with this new functionality I'm going to start work on a version that can read this file from a database. Also an admin web front end might be an idea? If you have any comments please email me.
One thing I have done from the outset is to parse the config file with an eye to making it extensible. So all tag get parsed into hashes, although this may make any module designed for parsing the config file a bit more complitacted, it does allow for custom subrouties to be written for the shipping code generator module.
revision 0.0.6
<shipd_conf 0.0.6>
<version 200071300>
<cache file="/etc/mivend/shipd_config_cache" cache="0|1">
<shipd_def company="some_company">
<courier UPS>
Quantifier = q
Max_q = 10000
Max_w = 10000
Max_box_q = 100
Max_box_w = 500
zone_pref0 = zone0
zone_pref1 = zone2
zone_pref2 = zone1
Cost_method = zones
zone0 = 3.99
zone1 = 4.99
zone2 = 5.99
default = 10
zonedb_engine = tab
zonedb = /etc/minvend/shipd/ups_zones
</courier>
# UPs delivery. The order is sorted by quanity "Quantifier = q"
# has a total max weight of 10000 units "Max_q = 10000" and
# a total quantity of 10000 units "Max_w = 10000". Each box
# can have a max of 100 units "Max_box_q = 100" or weight of
# 500 units "Max_box_w = 500".
# A delivery to destination A is looked up in a table
# to see what zone it is in. If it is not found in the
# table then the zone is set to default.
# The cost of the item is returned eg 5.99 if A is in zone
# zone2
# the file /etc/minvend/shipd/ups_zones is tab separated and
# looks like this...
# detination zone0 zone1 zone2
# usa 1 0 0
# london 0 1 1
# malta 0 1 0
# zone0 might be the new world
# zone1 might bethe europe
# zone2 might be eu.
# if a destination is in more than one zone then
# the zone_pref decides the zone. ie for a
# desination of london the zone would be
# "zone2" (London is in the eu which
# is in europe but has diferent taxes
# etc than outside the eu but in europe)
<courier PAT>
Quantifier = q
Max_q = 10000
Max_w = 10000
Max_box_q = 100
Max_box_w = 500
Cost_method = file_tab
File_location = /etc/minvend/shipd/PATs_costs
default = 100
# looks up the destination
# up in a two culunm table
# destination,price.
# If it finds the key
# the price is return
# else 100 is returned.
</courier>
<courier Bad_mail>
Quantifier = q
Max_q = 10000
Max_w = 10000
Max_box_q = 100
Max_box_w = 500
zone_def0 = bm1
zone_def1 = bm2
zone_def2 = bm3
zone_pref0 = bm3
zone_pref1 = bm1
zone_pref2 = bm2
key = code
zonedb_engine = Postgres
zonedb = all_info:faye:qwert:5432
zonetable = all_zones
Cost_method = zones
bm1 = 10
bm2 = 15
bm3 = 50
default = 200
# In this example all the zone definitions
# for many couriers are kept in a single
# table "all_zones" in a database "all_info"
# run by postgresql. For this courier we are
# only interested in the fields bm3,bm2,bm1
# which must be boolean. The feild "code"
# contains the destinations.
# If we find a zone then the cost is returned
# via the small zone table above.
</courier>
<courier Badder_mail>
Quantifier = q
Max_q = 10000
Max_w = 10000
Max_box_q = 100
Max_box_w = 500
Cost_method = function
function = 10 * __Eb__
# the cost is just 10 times
# the number of boxes.
# others variables include
# __Ew__ total weight of all boxes.
# __zonex__ zone price
</courier>
<courier Badder_mail>
Quantifier = q
Max_q = 10000
Max_w = 10000
Max_box_q = 100
Max_box_w = 500
zone_def0 = berm1
zone_def1 = berm2
zone_def2 = berm3
zone_pref0 = berm3
zone_pref1 = berm1
zone_pref2 = berm2
key = code
zonedb_engine = Postgres
zonedb = all_info:faye:qwert:5432
zonetable = all_zones
berm1 = 1
berm1 = 5
berm1 = 10
default = 100
Cost_method = function,zones
function = __zonex__ * __Eb__
# the cost is just 10 times
# the number of boxes.
# others variables include
# __Ew__ total weight of all boxes.
# __zonex__ zone price
# __zonex__ is substituted with the
# values coresponding to the
# zones the destinatio is in.
</courier>
</shipd_def>
This is a first draft of the server, it is goingto be refine before we put it into production. And I haven't shown you the db that contains all the info. However this should be enough to give you a jist.
I've now had the server up and running for two weeks with out a crash on my linux box :-), Jul 11 2000