Morning all

We've written an application for an auction house so that they can upload
their catalogues.

One of the things they need to be able to do is upload large zip files of
images.  Our app then unzips the file, resizes all the images to make
thumbnails and larger images, etc, using GD.

The problem is that they want to upload 50 meg files that contain up to 300
images.

We've got a dedicated server and we've increased the max post size, file
upload size and script execution time for php but that just kills the
server.

We're using http to upload the files at the moment.

So, a couple of questions:

1.  Would using ftp through php be more efficient.  I'm assuming that the
limitation is with the time taken to perform the task, not the fact that
we're using http or ftp as the protocol to perform the updload.

2.  Is the only realistic option going to be to tell them to break the
batches down into smaller chunks?

3.  Is there another answer?

Andy ideas gratefully received!

-- 
Andy Jacobs
http://www.redcatmedia.co.uk

On Sat, 18 Oct 2008 08:13:44 +0100, Andy Jacobs
 wrote in
<C51F4B38.137F6%nospam@redcatgroup.co.uk>:

>So, a couple of questions:
>
>1.  Would using ftp through php be more efficient.  I'm assuming that the
>limitation is with the time taken to perform the task, not the fact that
>we're using http or ftp as the protocol to perform the updload.

I am not sure what you mean by "using ftp through php". FTP is a
protocol so you need an FTP server at one end and an FTP client at the
other - I don't see where php enters into this. Although most web
browsers have enough FTP support to retrieve files, they do not all have
the ability to use the FTP 'put' command to upload.

I have heard that IE can do FTP uploads but I rarely use IE and have
never looked for or used that feature.

>2.  Is the only realistic option going to be to tell them to break the
>batches down into smaller chunks?

I would look at that differently. If the objective is to upload the
images with few manual steps then uploading all the images with a single
action is the goal and that need not involve putting them all into one
file.

>3.  Is there another answer?

It depends on whether or not the client is willing and able to use tools
other than a web browser. If so then there are various options.

One that may seem a bit odd but might be relatively easy to understand
would be to use Subversion. For the server end you can use a web server
with WEBDAV - over TLS/SSL if you need the security. At the client end
there are various tools that could be used but assuming that the client
is a Windows user, TortoiseSVN might be the easiest since it integrates
into Windows explorer. I use other tools myself but it seems to be
popular with people who do not want to get too deep into how things
work.

If the purpose of the zip file was just to do the upload then you can
replace [put images in folder, add to zip, upload zip] with [put images
in folder, add to version control, commit] so once you get used to it,
it should be just a easy.

Since you are not really looking for the version control features of
Subversion you may want to investigate other ways to synchronize
directories.

-- 
Owen Rees
[one of] my preferred email address[es] and more stuff can be
found at <http://www.users.waitrose.com/~owenrees/index.html>

On 18/10/08 12:11, in article dkdjf49aj82gckc54obt7djkhr1vgjsa9k@4ax.com,
"Owen Rees"  wrote:

> On Sat, 18 Oct 2008 08:13:44 +0100, Andy Jacobs
>  wrote in
> <C51F4B38.137F6%nospam@redcatgroup.co.uk>:
> 
>> So, a couple of questions:
>> 
>> 1.  Would using ftp through php be more efficient.  I'm assuming that the
>> limitation is with the time taken to perform the task, not the fact that
>> we're using http or ftp as the protocol to perform the updload.
> 
> I am not sure what you mean by "using ftp through php". FTP is a
> protocol so you need an FTP server at one end and an FTP client at the
> other - I don't see where php enters into this. Although most web
> browsers have enough FTP support to retrieve files, they do not all have
> the ability to use the FTP 'put' command to upload.

ftp_connect($HOST);

To be honest, I think we're going to have the same problem.  If the max
execution time of a php script is 60 seconds then it isn't going to matter
whether we're using ftp or http, it's still going to time out unless we set
a large max_execution_time variable, in which case the server grinds to a
halt with php.cgi using up huge chunks of CPU.

-- 
Andy Jacobs
http://www.redcatmedia.co.uk

On Sat, 18 Oct 2008 08:13:44 +0100, Andy Jacobs put finger to keyboard
and typed:
>
>3.  Is there another answer?

The most sensible option is for them to use another method of
uploading. Can't you just create an FTP account on the server for
them, and get them to upload the files to that using something like
FileZilla or Coffee Cup FTP?

Mark
-- 
"There must be a place, under the sun, where hearts of olden 
glory grow young"
http://mark.goodge.co.uk - my pointless blog
http://www.good-stuff.co.uk - my less pointless stuff

On Sat, 18 Oct 2008 19:31:33 +0100, Andy Jacobs put finger to keyboard
and typed:

>On 18/10/08 12:11, in article dkdjf49aj82gckc54obt7djkhr1vgjsa9k@4ax.com,
>"Owen Rees"  wrote:
>
>> On Sat, 18 Oct 2008 08:13:44 +0100, Andy Jacobs
>>  wrote in
>> <C51F4B38.137F6%nospam@redcatgroup.co.uk>:
>> 
>>> So, a couple of questions:
>>> 
>>> 1.  Would using ftp through php be more efficient.  I'm assuming that the
>>> limitation is with the time taken to perform the task, not the fact that
>>> we're using http or ftp as the protocol to perform the updload.
>> 
>> I am not sure what you mean by "using ftp through php". FTP is a
>> protocol so you need an FTP server at one end and an FTP client at the
>> other - I don't see where php enters into this. Although most web
>> browsers have enough FTP support to retrieve files, they do not all have
>> the ability to use the FTP 'put' command to upload.
>
>ftp_connect($HOST);
>
>To be honest, I think we're going to have the same problem.  If the max
>execution time of a php script is 60 seconds then it isn't going to matter
>whether we're using ftp or http, it's still going to time out unless we set
>a large max_execution_time variable, in which case the server grinds to a
>halt with php.cgi using up huge chunks of CPU.

If they upload via FTP, and the script then runs as a CLI (instead of
being called by the web server as CGI), it can be set to a low
priority and given as much time as it needs to do the work without
slowing down the rest of the server. Unless they need to see the
results instantly, it can even be set to run overnight or something.

Mark 
-- 
"There must be a place, under the sun, where hearts of olden 
glory grow young"
http://mark.goodge.co.uk - my pointless blog
http://www.good-stuff.co.uk - my less pointless stuff

On Sat, 18 Oct 2008 08:13:44 +0100, Andy Jacobs wrote:
[...]
> We've got a dedicated server and we've increased the max post size, file
> upload size and script execution time for php but that just kills the
> server.
>
> We're using http to upload the files at the moment.

Just don't use PHP - it has the habit of keeping the whole data in memory.

You can certainly achieve the desired behaviour when using something more
sensible like Python CGI (but any other reasonable programming language
should do as well) which doesn't necessarily keep the file upload data in
memory. I have implemented a file upload application a few years ago and
file sizes were only limited by the available disk space.

BTW, one limitation of http uploads (via POST) is that you can't resume
interrupted uploads.

WebDAV might be a better option.


Christof

-- 
http://cmeerw.org                              sip:cmeerw at cmeerw.org
mailto:cmeerw at cmeerw.org                   xmpp:cmeerw at cmeerw.org

On Sat, 18 Oct 2008 19:31:33 +0100, Andy Jacobs
 wrote in
<C51FEA15.13804%nospam@redcatgroup.co.uk>:

>ftp_connect($HOST);
>
>To be honest, I think we're going to have the same problem.  If the max
>execution time of a php script is 60 seconds then it isn't going to matter
>whether we're using ftp or http, it's still going to time out unless we set
>a large max_execution_time variable, in which case the server grinds to a
>halt with php.cgi using up huge chunks of CPU.

I was under the impression that the problem was in getting the large
volume of data from a customer system acting as a web client to the
system running the web server. If you then want to send it on somewhere
else by FTP I can see how the PHP FTP functions could be used but I did
not see that requirement in the original description of the problem.

-- 
Owen Rees
[one of] my preferred email address[es] and more stuff can be
found at <http://www.users.waitrose.com/~owenrees/index.html>

On 18/10/08 21:54, in article
6tikf49tvrmo5o17k4hc0hmmli92710dd2@news.markshouse.net, "Mark Goodge"
 wrote:

> On Sat, 18 Oct 2008 08:13:44 +0100, Andy Jacobs put finger to keyboard
> and typed:
>> 
>> 3.  Is there another answer?
> 
> The most sensible option is for them to use another method of
> uploading. Can't you just create an FTP account on the server for
> them, and get them to upload the files to that using something like
> FileZilla or Coffee Cup FTP?

I think this is going to be the route we go down.

Then we just need to run a script to loop through the directory and resize
all the images.

-- 
Andy Jacobs
http://www.redcatmedia.co.uk

Andy Jacobs wrote:

> On 18/10/08 21:54, in article
> 6tikf49tvrmo5o17k4hc0hmmli92710dd2@news.markshouse.net, "Mark Goodge"
>  wrote:
>
> > On Sat, 18 Oct 2008 08:13:44 +0100, Andy Jacobs put finger to keyboard
> > and typed:
> >>
> >> 3.  Is there another answer?
> >
> > The most sensible option is for them to use another method of
> > uploading. Can't you just create an FTP account on the server for
> > them, and get them to upload the files to that using something like
> > FileZilla or Coffee Cup FTP?
>
> I think this is going to be the route we go down.
>
> Then we just need to run a script to loop through the directory and resize
> all the images.
>
Hi Andy.
I think that you would be wise to get your clients to re-size the
images before placeing them on the web. I know, some people are not as
good with a lot of  this stuff.
But the quality of the images will not be as good on a Windows box if
you re-size with the browser. Cause Windows does a shit job of re-
sizeing images.
BTW I like the  new site, however a few of those pages with mainly
text could perhaps do with some images, just to break the text up a
bit.
--
Regards Chad. http://freewebdesignonline.org

<uk.net.web.authoring>
<Chaddy2222>
<Sun, 19 Oct 2008 08:12:29 -0700 (PDT)>


> BTW I like the  new site
> 

Seen worse - but that light tan/fawn colour along with a almost aqua 
blue colour for the nav bar links sucks .

Also , The brown bar at the top of the page is quite pointless and looks 
awful .


-- 
www.phpguestbook.co.uk/phpgb
(the best php guestbook on planet earth)

On Sun, 19 Oct 2008 08:12:29 -0700, Chaddy2222 wrote:
> I think that you would be wise to get your clients to re-size the images
> before placeing them on the web. I know, some people are not as good
> with a lot of  this stuff.
> But the quality of the images will not be as good on a Windows box if
> you re-size with the browser. Cause Windows does a shit job of re-
> sizeing images.

Just to be pedantic, it's not Windows but the browser that does the 
resizing. Most web browsers don't do proper interpolation when resizing 
pictures because it takes too long.

The other reason for uploading images correct size is loading time, 
especially if the picture is a big file and ends up displayed small. With 
broadband connections the difference isn't as notceable as it used to be 
on dialup, but even so, bandwidth costs money and time.

-- 
Anahata
anahata@treewind.co.uk -+- http://www.treewind.co.uk
Home: 01638 720444         Mob: 07976 263827

On 19/10/08 16:12, in article
54917cda-b324-43e9-861b-88e7cf7ff0f3@v13g2000pro.googlegroups.com,
"Chaddy2222"  wrote:

> 
> 
> Andy Jacobs wrote:
> 
>> On 18/10/08 21:54, in article
>> 6tikf49tvrmo5o17k4hc0hmmli92710dd2@news.markshouse.net, "Mark Goodge"
>>  wrote:
>> 
>>> On Sat, 18 Oct 2008 08:13:44 +0100, Andy Jacobs put finger to keyboard
>>> and typed:
>>>> 
>>>> 3.  Is there another answer?
>>> 
>>> The most sensible option is for them to use another method of
>>> uploading. Can't you just create an FTP account on the server for
>>> them, and get them to upload the files to that using something like
>>> FileZilla or Coffee Cup FTP?
>> 
>> I think this is going to be the route we go down.
>> 
>> Then we just need to run a script to loop through the directory and resize
>> all the images.
>> 
> Hi Andy.
> I think that you would be wise to get your clients to re-size the
> images before placeing them on the web. I know, some people are not as
> good with a lot of  this stuff.
> But the quality of the images will not be as good on a Windows box if
> you re-size with the browser. Cause Windows does a shit job of re-
> sizeing images.
> BTW I like the  new site, however a few of those pages with mainly
> text could perhaps do with some images, just to break the text up a
> bit.
> --
> Regards Chad. http://freewebdesignonline.org

We use GD to resize the images, not the browser.

Thanks for the heads up on the site.  It's only a stop gap.

-- 
Andy Jacobs
http://www.redcatmedia.co.uk

On 18 Oct, 09:13, Andy Jacobs  wrote:
> Morning all
>
> We've written an application for an auction house so that they can upload
> their catalogues.
>
> One of the things they need to be able to do is upload large zip files of
> images.  Our app then unzips the file, resizes all the images to make
> thumbnails and larger images, etc, using GD.
>
> The problem is that they want to upload 50 meg files that contain up to 300
> images.
>
> We've got a dedicated server and we've increased the max post size, file
> upload size and script execution time for php but that just kills the
> server.
>
> We're using http to upload the files at the moment.
>
> So, a couple of questions:
>
> 1.  Would using ftp through php be more efficient.  I'm assuming that the
> limitation is with the time taken to perform the task, not the fact that
> we're using http or ftp as the protocol to perform the updload.
>
> 2.  Is the only realistic option going to be to tell them to break the
> batches down into smaller chunks?
>
> 3.  Is there another answer?

I wouldn't link through PHP. You can put a direct ftp:// link as an
anchor tag if you need to. Alternatively provide an ftp/webdav tool.

However, if you want to keep it in the website, then there are Java
(not JS) applets available that should help and can manage some of the
uploading including things like autoresume and uploading a complete
directory in one go. I'd also split the task in two. Firstly, get the
images uploaded, then secondly resize - probably using a cron job.

In general though I would encourage the client to resize first - it'll
save a lot of time and bandwidth. We use Faststone Image Resizer which
enables a batch of files to be resized en bloc and is pretty easier to
use with flexibility for intelligent resizing.


Saul
www.notanant.com
communities of websites