Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Gaia Import
#1
I would like to import subsets of GAIA data into SkyTools.

The GAIA Archive is located here: http://cdn.gea.esac.esa.int/Gaia/gaia_source/csv/

The data sets are large, so I have downloaded and created small subsets of less than 1000 stars per file set. I do not see an import facility in SKT3Pro that would allow me to perform the import.

Can you provide guidance.

Will SKT4Pro provide this facility assuming SKT3Pro does not?

Thank you!
Reply
#2
Hi,

The SkyTools star database is integrated, meaning that stars from various catalogs are all put together to create the star database, taking care that each star only appears once, has the best available data of all types, and is fully cross referenced. This includes multiple star components. So the proper way to add the Gaia catalog will be to fully rebuild the star database from scratch, including the Gaia data. The DR1 data release was too recent for this to have been done yet, even for SkyTools 4, and it turns out that DR1 isn't really of high enough quality, so I am awaiting DR2. Maybe for ST5, depending on when the data is released. I expect it to be a game changer, particularly for double star observing!

The current database of stars to 20th magnitude is still pretty darned good for the primary purposes of SkyTools (planning and finding).
Clear skies,
Greg

SkyTools Developer
Reply
#3
(2017-11-19, 08:17 PM)theskyhound Wrote: Hi,

The SkyTools star database is integrated, meaning that stars from various catalogs are all put together to create the star database, taking care that each star only appears once, has the best available data of all types, and is fully cross referenced. This includes multiple star components. So the proper way to add the Gaia catalog will be to fully rebuild the star database from scratch, including the Gaia data. The DR1 data release was too recent for this to have been done yet, even for SkyTools 4, and it turns out that DR1 isn't really of high enough quality, so I am awaiting DR2. Maybe for ST5, depending on when the data is released. I expect it to be a game changer, particularly for double star observing!

The current database of stars to 20th magnitude is still pretty darned good for the primary purposes of SkyTools (planning and finding).

Will SKT4Pro provide this facility?
Reply
#4
(2017-11-19, 10:57 PM)Starmaster Wrote:
(2017-11-19, 08:17 PM)theskyhound Wrote: Hi,

The SkyTools star database is integrated, meaning that stars from various catalogs are all put together to create the star database, taking care that each star only appears once, has the best available data of all types, and is fully cross referenced. This includes multiple star components. So the proper way to add the Gaia catalog will be to fully rebuild the star database from scratch, including the Gaia data. The DR1 data release was too recent for this to have been done yet, even for SkyTools 4, and it turns out that DR1 isn't really of high enough quality, so I am awaiting DR2. Maybe for ST5, depending on when the data is released. I expect it to be a game changer, particularly for double star observing!

The current database of stars to 20th magnitude is still pretty darned good for the primary purposes of SkyTools (planning and finding).

Will SKT4Pro provide this facility?

What facility? I don't understand what you are asking.
Clear skies,
Greg

SkyTools Developer
Reply
#5
(2017-11-19, 11:11 PM)theskyhound Wrote:
(2017-11-19, 10:57 PM)Starmaster Wrote:
(2017-11-19, 08:17 PM)theskyhound Wrote: Hi,

The SkyTools star database is integrated, meaning that stars from various catalogs are all put together to create the star database, taking care that each star only appears once, has the best available data of all types, and is fully cross referenced. This includes multiple star components. So the proper way to add the Gaia catalog will be to fully rebuild the star database from scratch, including the Gaia data. The DR1 data release was too recent for this to have been done yet, even for SkyTools 4, and it turns out that DR1 isn't really of high enough quality, so I am awaiting DR2. Maybe for ST5, depending on when the data is released. I expect it to be a game changer, particularly for double star observing!

The current database of stars to 20th magnitude is still pretty darned good for the primary purposes of SkyTools (planning and finding).

Will SKT4Pro provide this facility?

What facility? I don't understand what you are asking.

Will SKT4Pro provide an import utility for .csv files? I would like to add data beyond mag 20 in sets of 500-1000 records at a time. I am not interested in combining data with your set, rather I would prefer to plot these stars independently of your integrated data.

If not I understand and will move on...
Reply
#6
(2017-11-20, 06:30 PM)Starmaster Wrote:
(2017-11-19, 11:11 PM)theskyhound Wrote:
(2017-11-19, 10:57 PM)Starmaster Wrote:
(2017-11-19, 08:17 PM)theskyhound Wrote: Hi,

The SkyTools star database is integrated, meaning that stars from various catalogs are all put together to create the star database, taking care that each star only appears once, has the best available data of all types, and is fully cross referenced. This includes multiple star components. So the proper way to add the Gaia catalog will be to fully rebuild the star database from scratch, including the Gaia data. The DR1 data release was too recent for this to have been done yet, even for SkyTools 4, and it turns out that DR1 isn't really of high enough quality, so I am awaiting DR2. Maybe for ST5, depending on when the data is released. I expect it to be a game changer, particularly for double star observing!

The current database of stars to 20th magnitude is still pretty darned good for the primary purposes of SkyTools (planning and finding).

Will SKT4Pro provide this facility?

What facility? I don't understand what you are asking.

Will SKT4Pro provide an import utility for .csv files? I would like to add data beyond mag 20 in sets of 500-1000 records at a time. I am not interested in combining data with your set, rather I would prefer to plot these stars independently of your integrated data.

If not I understand and will move on...

Hi,

I apologize for not being more clear. There is no facility for adding large numbers of stars to SkyTools, nor is there likely to be in the future. Adding large numbers of stars without full integration would make the charts unbearably slow. Instead, I plan to rebuild the entire SkyTools star database, incorporating Gaia DR2 star data.
Clear skies,
Greg

SkyTools Developer
Reply
#7
(2017-11-20, 08:15 PM)theskyhound Wrote:
(2017-11-20, 06:30 PM)Starmaster Wrote:
(2017-11-19, 11:11 PM)theskyhound Wrote:
(2017-11-19, 10:57 PM)Starmaster Wrote:
(2017-11-19, 08:17 PM)theskyhound Wrote: Hi,

The SkyTools star database is integrated, meaning that stars from various catalogs are all put together to create the star database, taking care that each star only appears once, has the best available data of all types, and is fully cross referenced. This includes multiple star components. So the proper way to add the Gaia catalog will be to fully rebuild the star database from scratch, including the Gaia data. The DR1 data release was too recent for this to have been done yet, even for SkyTools 4, and it turns out that DR1 isn't really of high enough quality, so I am awaiting DR2. Maybe for ST5, depending on when the data is released. I expect it to be a game changer, particularly for double star observing!

The current database of stars to 20th magnitude is still pretty darned good for the primary purposes of SkyTools (planning and finding).

Will SKT4Pro provide this facility?

What facility? I don't understand what you are asking.

Will SKT4Pro provide an import utility for .csv files? I would like to add data beyond mag 20 in sets of 500-1000 records at a time. I am not interested in combining data with your set, rather I would prefer to plot these stars independently of your integrated data.

If not I understand and will move on...

Hi,

I apologize for not being more clear. There is no facility for adding large numbers of stars to SkyTools, nor is there likely to be in the future. Adding large numbers of stars without full integration would make the charts unbearably slow. Instead, I plan to rebuild the entire SkyTools star database, incorporating Gaia DR2 star data.

Cry
Reply
#8
Well, I am sorry you feel that way, but the integrated databases are in fact one of the SkyTools features that I a most proud of. It is relatively easy to build a data engine that simply piles data on from raw catalogs, but there are all sorts of problems with that. You aren't going to get a double star database like SkyTools has that way, for instance.
Clear skies,
Greg

SkyTools Developer
Reply
#9
(2017-11-21, 05:39 AM)theskyhound Wrote: Well, I am sorry you feel that way, but the integrated databases are in fact one of the SkyTools features that I a most proud of. It is relatively easy to build a data engine that simply piles data on from raw catalogs, but there are all sorts of problems with that. You aren't going to get a double star database like SkyTools has that way, for instance.

I've read the history on the Yahoo Forums of how you refined your catalog, and for its intended use it is an excellent effort that you've made.

But it is a walled garden, stuck in time, and the community is dependent on you for its update. 

This is a  fact and is not meant as a negative comment about you personally.

New catalogs are generated constantly and the ability to import those catalogs and use them in sky charting tools, (FLAWS Included),  is an important factor in the selection of a charting tool for purposes which go well beyond your intended use cases.

Given your response I am disappointed by your decision, but I am sure my requirements are atypical of the user community for Sky Tools. 

In any event it is a small matter of minimal consequence. I accept your decision and will seek a solution elsewhere. 

Kind regards...
Reply
#10
(2017-11-21, 05:18 PM)Starmaster Wrote: New catalogs are generated constantly and the ability to import those catalogs and use them in sky charting tools, (FLAWS Included),  is an important factor in the selection of a charting tool for purposes which go well beyond your intended use cases.

It would be very helpful to know what these purposes are.

Obviously I feel that the work that I have put into integrating the database is well worth it. Flaws are not a good thing. Having the same star show up in three different places depending on the catalog displayed is not a good thing. Not having access to the data from one catalog, such as photometry, while displaying another, is not a good thing. Zooming in on a double star and seeing five different B components, two of which were observations from the 19th century (when the pair appeared differently due to proper motion) is not a good thing.

If SkyTools was used for, say, astrometry I would have to agree with your sentiments. You would need access to the latest and most accurate astrometric data. But you cannot do astrometry with SkyTools. For the reasonable purposes of SkyTools, the current star catalog is just fine. It still has the only fully integrated double star catalog, and I believe it is still one of the best corrected versions of the WDS data. Very few amateurs image deep enough that they need a catalog that goes deeper than 20th magnitude.

I wish people reading this to understand that I cannot simply set aside the integrated database on a whim. Databases are complex. One might as well be telling me that my house should not have been made out of wood, because brick would have been a better material. Not only do I disagree completely, but I am not about to somehow replace all the wood with brick. We simply don't build brick houses here.

Integrating the Gaia data will likely take a year or so after the DR2 release, which is slated for April 2018. Starting with release of ST4, SkyTools will be updated yearly, like clockwork. There will be no more multi-year delays.

One thing you never told me was why you wanted to add Gaia stars on top of the existing database. Perhaps it would be helpful to explain what your purpose is. Maybe there is another solution.
Clear skies,
Greg

SkyTools Developer
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)