Hello webservice users,
The DMC has updated it's ws-bulkdataselect service: http://www.iris.edu/ws/bulkdataselect/
The new version adds two optional query parameters:
minimumlength
Enforce minimum segment length (seconds).
Only time-series segments of this length or longer will be returned.
longestonly
If specified, only the longest continuous segment for each channel is
returned.
These new parameters can be used to reduce or avoid receiving data containing gaps.
The command line FetchBulkData client script that uses this service to request data has been updated with options to specify these new parameters, the latest client scripts are available here:
http://www.iris.edu/ws/wsclients/
Usage details: like the quality parameter, the two new parameters may be specified in the main selection list or separately as post parameters.
For example, this selection will return only the longest data, that is at least one day (86400 seconds) long:
minimumlength 86400.0
longestonly
TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00
regards
Web services team.
-
Hi Chad
I guess it is only natural that as these services are new, there will
be some changes. It looks like this one is backwards compatible, but
the previous one to the station web service wasn't and would cause
problems for any other clients out there.
Have you all given any thought to a longer term versioning for these
systems? Here is the problem I worry about. I put code into a software
package to access these web services. My code is obviously tuned to
the way the web service works at the time I write and test it. Later,
after I have made a release, IRIS makes a non-backwards compatible
change. Then, my client breaks and it looks to the rest of the world
like I have written buggy code and it can't be fixed without a new
release of my client. I also have little to no forewarning that the
change is coming, and therefore have little ability to be proactive in
what I do.
If you change the IRIS web page, people just notice the change and
figure things out. They may like or dislike the change, but they can
adapt. Web services are fundamentally different than web pages. A
small change that would cause no problem for a person can easily cause
a web service client to completely fail.
I guess it feels to me like once you put out a web service that will
be called by third party clients, you have some obligation to either
make only backward compatible changes, or to run the old version and
the new version in parallel for some time to allow an orderly
transition. Given the development cycle for clients, my guess is this
might be closer to years than weeks. Although the IDL revision for
fissures/dhi was never implemented, we addressed this issue in DHI by
encoding the IDL version number into the name service lookup system,
so an IDL 1.0 client would be able to find IDL1.0 servers and would
not accidentally try to connect to IDL 2.0 servers. Obviously web
services will use different mechanisms, but if they are both going to
be used and to evolve at the same time, then you have to provide some
means for the communications protocol contract to remain stable.
Otherwise every change, even small ones, can and will royally screw
your users.
This versioning could easily be accomplished in several ways, either
putting a version number in the url, say
http://www.iris.edu/ws/1.0/bulkdataselect/
or as a parameter
http://www.iris.edu/ws/bulkdataselect/?version=1.0
The former has they advantage of you leaving the old server up and
just adding new ones, while the second allows you to be fancy in your
implementation to tailor the output to the client. I guess I prefer
the stability of the first, but as a client writer I can deal with
either.
thanks,
Philip
On Wed, Apr 20, 2011 at 7:30 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:
Hello webservice users,
The DMC has updated it's ws-bulkdataselect service:
http://www.iris.edu/ws/bulkdataselect/
The new version adds two optional query parameters:
minimumlength
Enforce minimum segment length (seconds).
Only time-series segments of this length or longer will be returned.
longestonly
If specified, only the longest continuous segment for each channel is
returned.
These new parameters can be used to reduce or avoid receiving data
containing gaps.
The command line FetchBulkData client script that uses this service to
request data has been updated with options to specify these new parameters,
the latest client scripts are available here:
http://www.iris.edu/ws/wsclients/
Usage details: like the quality parameter, the two new parameters may be
specified in the main selection list or separately as post parameters.
For example, this selection will return only the longest data, that is at
least one day (86400 seconds) long:
minimumlength 86400.0
longestonly
TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00
regards
Web services team.
_______________________________________________
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices
-
Hi Philip,
We have discussed how to deal with versioned services/results including both of the suggestions you make below. We landed at using different base URLs for major changes in API or content, either adding "v2" or changing the service name.
There is a balance to be struck between having numerous versions and too little versioning. I disagree that "every change, even small ones, can and will royally screw your users". Adding a parameter to a service and nothing else changes does not, in my opinion, warrant a completely new URL. Likewise, adding a completely new element to XML results without changing the structure of the existing elements does also not warrant a 99% parallel service, wouldn't be very eXtensible if adding details broke a parser.
For the recent update here was the logic. The change was relatively disruptive, we added a whole new <Network> level to the StationXML. StationXML is not quite settled but is getting very close, so we don't expect this to happen much more often if at all. The ws-station service is relatively new, most folks are using it with the Fetch* scripts we provide which were not effected because of the stream processing of XML (i.e. you are one of the few that are parsing it directly) so that limits the disruption. In short, the impact of changing the results versus changing the URL in this case was not worth having a new service. If, say, a year from now we need to do that big of a change and the adoption of ws-station is high we would probably make a new version.
StationXML will continue to evolve to match the needs of SCEDC, NCEDC, IRIS and USGS (and maybe the FDSN in the future). At this point we do not expect large structural changes, but we do expect additions which should not break any real XML parser.
Chad
On Apr 22, 2011, at 6:51 AM, Philip Crotwell wrote:
Hi Chad
I guess it is only natural that as these services are new, there will
be some changes. It looks like this one is backwards compatible, but
the previous one to the station web service wasn't and would cause
problems for any other clients out there.
Have you all given any thought to a longer term versioning for these
systems? Here is the problem I worry about. I put code into a software
package to access these web services. My code is obviously tuned to
the way the web service works at the time I write and test it. Later,
after I have made a release, IRIS makes a non-backwards compatible
change. Then, my client breaks and it looks to the rest of the world
like I have written buggy code and it can't be fixed without a new
release of my client. I also have little to no forewarning that the
change is coming, and therefore have little ability to be proactive in
what I do.
If you change the IRIS web page, people just notice the change and
figure things out. They may like or dislike the change, but they can
adapt. Web services are fundamentally different than web pages. A
small change that would cause no problem for a person can easily cause
a web service client to completely fail.
I guess it feels to me like once you put out a web service that will
be called by third party clients, you have some obligation to either
make only backward compatible changes, or to run the old version and
the new version in parallel for some time to allow an orderly
transition. Given the development cycle for clients, my guess is this
might be closer to years than weeks. Although the IDL revision for
fissures/dhi was never implemented, we addressed this issue in DHI by
encoding the IDL version number into the name service lookup system,
so an IDL 1.0 client would be able to find IDL1.0 servers and would
not accidentally try to connect to IDL 2.0 servers. Obviously web
services will use different mechanisms, but if they are both going to
be used and to evolve at the same time, then you have to provide some
means for the communications protocol contract to remain stable.
Otherwise every change, even small ones, can and will royally screw
your users.
This versioning could easily be accomplished in several ways, either
putting a version number in the url, say
http://www.iris.edu/ws/1.0/bulkdataselect/
or as a parameter
http://www.iris.edu/ws/bulkdataselect/?version=1.0
The former has they advantage of you leaving the old server up and
just adding new ones, while the second allows you to be fancy in your
implementation to tailor the output to the client. I guess I prefer
the stability of the first, but as a client writer I can deal with
either.
thanks,
Philip
On Wed, Apr 20, 2011 at 7:30 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:
Hello webservice users,
_______________________________________________
The DMC has updated it's ws-bulkdataselect service:
http://www.iris.edu/ws/bulkdataselect/
The new version adds two optional query parameters:
minimumlength
Enforce minimum segment length (seconds).
Only time-series segments of this length or longer will be returned.
longestonly
If specified, only the longest continuous segment for each channel is
returned.
These new parameters can be used to reduce or avoid receiving data
containing gaps.
The command line FetchBulkData client script that uses this service to
request data has been updated with options to specify these new parameters,
the latest client scripts are available here:
http://www.iris.edu/ws/wsclients/
Usage details: like the quality parameter, the two new parameters may be
specified in the main selection list or separately as post parameters.
For example, this selection will return only the longest data, that is at
least one day (86400 seconds) long:
minimumlength 86400.0
longestonly
TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00
regards
Web services team.
_______________________________________________
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices
-
OK, sound like you have a plan for this. My concern was that the
station change was done with little advance warning and no grace
period. So, it broke my code. Luckily all the places it was used were
local, so I was able to quickly update them. As you say, this is all
new, so some instability is to be expected. And I am fine with
backward compatible changes (adding optional url args or extra xml
elements that can be ignored), those certainly don't need version
changes. But a change, like the station one, that was small in some
sense, yet it really altered the xml output in a pretty fundamental
way. I like the new stationxml better, so I am not complaining about
that. It is just that as a client writer, I am highly dependent on
those servers and at the mercy of the decisions the dmc makes while at
the same time having very little power or influence on how those
choices are made. Had I made a software release the day before you
pushed out the station change, I would have been SOL. And so I would
have expected that type of change to happen in beta, but for there to
be a grace period and some forewarning at least to the web services
email list after you declared these services ready for prime time.
At any rate, the new station ws is quite nice, so on the whole I am very happy.
Philip
On Fri, Apr 22, 2011 at 1:45 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:
Hi Philip,
We have discussed how to deal with versioned services/results including both of the suggestions you make below. We landed at using different base URLs for major changes in API or content, either adding "v2" or changing the service name.
There is a balance to be struck between having numerous versions and too little versioning. I disagree that "every change, even small ones, can and will royally screw your users". Adding a parameter to a service and nothing else changes does not, in my opinion, warrant a completely new URL. Likewise, adding a completely new element to XML results without changing the structure of the existing elements does also not warrant a 99% parallel service, wouldn't be very eXtensible if adding details broke a parser.
For the recent update here was the logic. The change was relatively disruptive, we added a whole new <Network> level to the StationXML. StationXML is not quite settled but is getting very close, so we don't expect this to happen much more often if at all. The ws-station service is relatively new, most folks are using it with the Fetch* scripts we provide which were not effected because of the stream processing of XML (i.e. you are one of the few that are parsing it directly) so that limits the disruption. In short, the impact of changing the results versus changing the URL in this case was not worth having a new service. If, say, a year from now we need to do that big of a change and the adoption of ws-station is high we would probably make a new version.
StationXML will continue to evolve to match the needs of SCEDC, NCEDC, IRIS and USGS (and maybe the FDSN in the future). At this point we do not expect large structural changes, but we do expect additions which should not break any real XML parser.
Chad
On Apr 22, 2011, at 6:51 AM, Philip Crotwell wrote:
Hi Chad
_______________________________________________
I guess it is only natural that as these services are new, there will
be some changes. It looks like this one is backwards compatible, but
the previous one to the station web service wasn't and would cause
problems for any other clients out there.
Have you all given any thought to a longer term versioning for these
systems? Here is the problem I worry about. I put code into a software
package to access these web services. My code is obviously tuned to
the way the web service works at the time I write and test it. Later,
after I have made a release, IRIS makes a non-backwards compatible
change. Then, my client breaks and it looks to the rest of the world
like I have written buggy code and it can't be fixed without a new
release of my client. I also have little to no forewarning that the
change is coming, and therefore have little ability to be proactive in
what I do.
If you change the IRIS web page, people just notice the change and
figure things out. They may like or dislike the change, but they can
adapt. Web services are fundamentally different than web pages. A
small change that would cause no problem for a person can easily cause
a web service client to completely fail.
I guess it feels to me like once you put out a web service that will
be called by third party clients, you have some obligation to either
make only backward compatible changes, or to run the old version and
the new version in parallel for some time to allow an orderly
transition. Given the development cycle for clients, my guess is this
might be closer to years than weeks. Although the IDL revision for
fissures/dhi was never implemented, we addressed this issue in DHI by
encoding the IDL version number into the name service lookup system,
so an IDL 1.0 client would be able to find IDL1.0 servers and would
not accidentally try to connect to IDL 2.0 servers. Obviously web
services will use different mechanisms, but if they are both going to
be used and to evolve at the same time, then you have to provide some
means for the communications protocol contract to remain stable.
Otherwise every change, even small ones, can and will royally screw
your users.
This versioning could easily be accomplished in several ways, either
putting a version number in the url, say
http://www.iris.edu/ws/1.0/bulkdataselect/
or as a parameter
http://www.iris.edu/ws/bulkdataselect/?version=1.0
The former has they advantage of you leaving the old server up and
just adding new ones, while the second allows you to be fancy in your
implementation to tailor the output to the client. I guess I prefer
the stability of the first, but as a client writer I can deal with
either.
thanks,
Philip
On Wed, Apr 20, 2011 at 7:30 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:
Hello webservice users,
_______________________________________________
The DMC has updated it's ws-bulkdataselect service:
http://www.iris.edu/ws/bulkdataselect/
The new version adds two optional query parameters:
minimumlength
Enforce minimum segment length (seconds).
Only time-series segments of this length or longer will be returned.
longestonly
If specified, only the longest continuous segment for each channel is
returned.
These new parameters can be used to reduce or avoid receiving data
containing gaps.
The command line FetchBulkData client script that uses this service to
request data has been updated with options to specify these new parameters,
the latest client scripts are available here:
http://www.iris.edu/ws/wsclients/
Usage details: like the quality parameter, the two new parameters may be
specified in the main selection list or separately as post parameters.
For example, this selection will return only the longest data, that is at
least one day (86400 seconds) long:
minimumlength 86400.0
longestonly
TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00
regards
Web services team.
_______________________________________________
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices
webservices mailing list
webservices<at>iris.washington.edu
http://www.iris.washington.edu/mailman/listinfo/webservices
-
-
-
Hello web service users,
New versions of the DMC's web service Fetch scripts have been released and are available here:
http://www.iris.edu/ws/wsclients/
The FetchBulkData script has been made more robust by breaking large requests into groups and restarting the download of a group when the connection has been interrupted. These features combined allow FetchBulkData to better handle very large data requests and to recover automatically from network connectivity problems or service interruptions.
The web page now also includes client usage documentation with examples of usage and description of key features.
Each script has improved documentation, minor fixes and improved reporting that will help the DMC and users of these scripts diagnose problems.
Users of these scripts are encouraged to keep their copy up to date.
regards,
web services team
On Apr 20, 2011, at 4:30 PM, Chad Trabant wrote:
Hello webservice users,
The DMC has updated it's ws-bulkdataselect service: http://www.iris.edu/ws/bulkdataselect/
The new version adds two optional query parameters:
minimumlength
Enforce minimum segment length (seconds).
Only time-series segments of this length or longer will be returned.
longestonly
If specified, only the longest continuous segment for each channel is
returned.
These new parameters can be used to reduce or avoid receiving data containing gaps.
The command line FetchBulkData client script that uses this service to request data has been updated with options to specify these new parameters, the latest client scripts are available here:
http://www.iris.edu/ws/wsclients/
Usage details: like the quality parameter, the two new parameters may be specified in the main selection list or separately as post parameters.
For example, this selection will return only the longest data, that is at least one day (86400 seconds) long:
minimumlength 86400.0
longestonly
TA A25A -- BHZ 2010-084T00:00:00 2010-091T00:00:00
TA A25A -- BHE 2010-084T00:00:00 2010-091T00:00:00
regards
Web services team.
-
Hello web service users,
A new revision of the DMC's FetchBulkData script has been released and is available here:
http://www.iris.edu/ws/wsclients/
The FetchBulkData script has been updated to include options for selecting data by geographic region in to two ways:
a) Users may specify a rectangular region using minimum and maximum latitude and/or longitude options
b) Users may specify a circular region by defining a point and a maximum radius, optionally a minimum radius may be specified to selected a limited distance range.
The documentation on the page above now includes an example of selecting data by region.
This version also fixes a bug for selecting multiple channels of data with different time windows, like is possible with a BREQ_FAST or selection file.
Users of this script are encouraged to keep their copy up to date.
regards,
web services team
-