Training Data
This notebook assumes you have the DC2 data downloaded
You will have to change directory paths
[1]:
# In case you need to point to pre-existing scarlet install
import sys
import deepdisc
[1]:
# Standard imports
import sys, os
import numpy as np
import time
import glob
import scarlet
import sep
import astropy.io.fits as fits
from astropy.wcs import WCS
from astropy.stats import gaussian_fwhm_to_sigma
from astropy.coordinates import SkyCoord
from scarlet.display import AsinhMapping
from astropy.nddata import Cutout2D
# Astrodet imports
import deepdisc.preprocessing.detection as detection
import deepdisc.preprocessing.process as process
from deepdisc.astrodet.hsc import get_tract_patch_from_coord, get_hsc_data
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import matplotlib
import matplotlib.pyplot as plt
# use a better colormap and don't interpolate the pixels
matplotlib.rc('image', cmap='gray', interpolation='none', origin='lower')
from skimage.util.shape import view_as_blocks
from matplotlib import colors
import pandas as pd
import h5py
import json
First, let’s test scarlet using one DC2 image. The DC2 image data is divided into “tracts” and “patches” on the sky. You can get the data here https://data.lsstdesc.org/.
You will need to change the directory paths below
[3]:
from deepdisc.preprocessing.get_data import get_cutout
filters = ['u','g','r','i','z','y']
#dirpath = '/home/g4merz/DC2/coadd-t3828-t3829/deepCoadd-results/'
dirpath = '/home/shared/hsc/DC2/raw_data/'
nblocks=16 #The number of cutouts per side of an image. 4k CCDs are too large to train with, so we reduce the size
No input catalog
If you don’t have an input catalog that contains object locations, no worries! The object detection can be done with sep a standard code for astronmical image processing.
But first, we need to grab the images we want to preprocess. The function get_cutout produces an image cutout from a larger “patch” and is specific to the file format of the data used in the tutorial. You will need to make your own custom function to get an image saved in a different format
[39]:
tract = 3828
patch='2,2'
sp = 18
cutout,datsm, _ = get_cutout(dirpath,tract=tract,patch=patch,sp=sp,nblocks=nblocks,filters=filters,plot=False, get_psf=False)
[40]:
plt.imshow(datsm[1],norm=colors.LogNorm())
[40]:
<matplotlib.image.AxesImage at 0x7ffe4c7d85e0>
Now we can run detection and segmentation on the cutout image
To produce labels for an image, use deepdisc.preprocessing.detection.run_scarlet. The main inputs you will need are an image, a list of filters, and a psf (either an image or a gaussian standard deviation). Check the API documentation for details.
This will produce a few intermediate objects that contain all of the relevant information. However, we format them into FITS files for ease of access. This is done using deeepdisc.preprocess.process.write_scarlet_results
[27]:
#This function is specific to the data format. You will need your own function to produce "datas" to input to run_scarlet
def generate_training_data_example(dirpath, tract, patch, sp, outdir, plot_image=False, plot_stretch_Q=False, plot_scene=False,
plot_likelihood=False, write_results=True):
"""
Parameters
----------
c : SkyCoord object
The ra, dec pointing (single or lists of pointings)
plot_image : bool
Whether or not to plot the image
plot_stretch_Q : bool
Whether or not to plot different normalizations of your image using the stretch, Q parameters.
plot_scene : bool
Whether or not plot scene with scarlet
plot_likelihood : bool
Whether or not plot the log likelihood of the scarlet fitting
write_results : bool
Whether or not to write results to FITS file
cutout_size : [int, int]
Cutout shape of image
Returns
-------
The scarlet image test in FITS files.
"""
print(tract,patch,sp)
print()
cutout,datas, _ = get_cutout(dirpath,tract=tract,patch=patch,sp=sp,nblocks=nblocks,filters=filters,plot=False, get_psf=False)
### Run scarlet on image ###
# Image pixel scale in arcsec/pixel
ps = 0.2
# Approximate PSF size
sigma_obs = gaussian_fwhm_to_sigma*0.8/ps
# Run Scarlet
out = detection.run_scarlet(datas, filters, catalog=None, lvl=2, sigma_model=1, sigma_obs=sigma_obs, psf=None, plot_scene=plot_scene,
max_chi2=1000000, morph_thresh=1, stretch=1, Q=5,
plot_wavelet=False, plot_likelihood=plot_likelihood, plot_sources=False, add_ellipses=False,
add_labels=False, add_boxes=False, lvl_segmask=2, maskthresh=0.005,return_models=False)
# Unpack output
observation, starlet_sources, model_frame, catalog, segmentation_masks = out
# Save Scarlet data to FITS file
if write_results:
filenames = process.write_scarlet_results_nomodels(datas, observation, starlet_sources, model_frame,
segmentation_masks, outdir=outdir,
filters=filters, s=f'{tract}_{patch}_{sp}', source_catalog=catalog)
print(f'\nSaved scarlet results as {filenames} \n')
[ ]:
%%time
outdir='/home/g4merz/ddtutorials/'
generate_training_data_example(dirpath, tract='3828',patch='1,2',sp=4,plot_scene=False, plot_likelihood=False, write_results=True, outdir=outdir)
Using an input catalog
The cells below assume you have an input catalog all_tracts_cat.csv corresponding to the tracts and patches you’ve downloaded. We can run the code without one, but it is necessary for truth-matching any quantities
[46]:
import astropy.units as u
from astropy.coordinates import SkyCoord
from astropy.stats import gaussian_fwhm_to_sigma
dall=pd.read_csv('/home/shared/hsc/DC2/raw_data/all_tracts_truth_cat.csv')
#dall=pd.read_csv('/home/g4merz/DC2/nersc_data/data/tracts_data/all_tracts_cat.csv')
ra_all = dall['ra'][:].values
dec_all = dall['dec'][:].values
allcatalog = SkyCoord(ra=ra_all*u.degree, dec=dec_all*u.degree)
[49]:
def get_cutout_cat(dirpath,dcat,skycoords, id_col, tract,patch,sp,nblocks=4,filters=['u','g','r','i','z','y']):
'''
WARNING!!!!!
It is not efficient to have the full catalog (defined here as dall) as input to a function when doing multiprocesing.
Keep it in the top level process
'''
cutout,datsm, psf = get_cutout(dirpath,tract=tract,patch=patch,sp=sp,nblocks=nblocks,filters=filters,plot=False, get_psf=True)
xs,ys = cutout.wcs.world_to_pixel(skycoords)
inds = np.where((xs>=0) & (xs<cutout.shape[1]-1) & (ys>=0) & (ys<cutout.shape[0]-1))[0]
dcut = dcat.iloc[inds]
dcut['new_x'] = xs[inds]
dcut['new_y'] = ys[inds]
#dcut = dcut[dcut['mag_i']<25.3]
column_to_move = dcut.pop(id_col)
# insert column with insert(location, column_name, column_value)
dcut.insert(0, id_col, column_to_move)
dcut.sort_values(by=id_col)
return datsm, dcut, psf
[50]:
datsm,dcut,psf = get_cutout_cat(dirpath, dall, allcatalog, 'id', '3828','1,2',4,nblocks=8)
/tmp/ipykernel_1865293/2061951149.py:16: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_x'] = xs[inds]
/tmp/ipykernel_1865293/2061951149.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_y'] = ys[inds]
[16]:
fig, ax = plt.subplots(1,1, figsize=(7,7))
#plt.figure(figsize=(7,7))
ax.imshow(datsm[3],norm=colors.LogNorm(),origin='lower')
ax.scatter(dcut['new_x'].values,dcut['new_y'].values,marker='.')
ax.axis('off')
#ax[1].imshow(datsm2[3],norm=colors.LogNorm(),origin='lower')
#ax[1].scatter(dcut2['new_x'].values,dcut2['new_y'].values,marker='.')
#ax[1].axis('off')
[16]:
(-0.5, 524.5, -0.5, 524.5)
[54]:
filters = ['u','g','r','i','z','y']
def generate_training_data_example(dirpath, tract, patch, sp, outdir, plot_image=False, plot_stretch_Q=False, plot_scene=False,
plot_likelihood=False, write_results=True):
"""
Parameters
----------
c : SkyCoord object
The ra, dec pointing (single or lists of pointings)
plot_image : bool
Whether or not to plot the image
plot_stretch_Q : bool
Whether or not to plot different normalizations of your image using the stretch, Q parameters.
plot_scene : bool
Whether or not plot scene with scarlet
plot_likelihood : bool
Whether or not plot the log likelihood of the scarlet fitting
write_results : bool
Whether or not to write results to FITS file
cutout_size : [int, int]
Cutout shape of image
Returns
-------
The scarlet image test in FITS files.
"""
datas,dcut,psf = get_cutout_cat(dirpath,dall,allcatalog,'id',tract=tract,patch=patch,sp=sp,nblocks=nblocks)
#cut = np.where(dcut['mag_i'].values<25.3)[0]
#dcut = dcut.iloc[cut]
#cutout,datas,psf = get_cutout(dirpath,tract=tract,patch=patch,sp=sp,nblocks=nb,filters=filters,plot=False)
### Run scarlet on image ###
# HSC pixel scale in arcsec/pixel
ps = 0.2
# Approximate PSF size in UD field according to HSC DR2 paper is 0.8 arcsec
sigma_obs = gaussian_fwhm_to_sigma*0.8/ps
#psf = np.load(f'/home/g4merz/DC2/nersc_data/data/psfs/{tract}_{patch}_0_psfs.npy')
# Run Scarlet
out = detection.run_scarlet(datas, filters, catalog=dcut, lvl=2, sigma_model=1, sigma_obs=sigma_obs, psf=psf, plot_scene=plot_scene,
max_chi2=1000000, morph_thresh=1, stretch=1, Q=5,
plot_wavelet=False, plot_likelihood=plot_likelihood, plot_sources=False, add_ellipses=False,
add_labels=False, add_boxes=False, lvl_segmask=2, maskthresh=0.005,return_models=False)
# Unpack output
#observation, starlet_sources, model_frame, catalog, catalog_deblended, segmentation_masks = out
observation, starlet_sources, model_frame, catalog, segmentation_masks = out
# Save Scarlet data to FITS file
if write_results:
filenames = process.write_scarlet_results_nomodels(datas, observation, starlet_sources, model_frame,
segmentation_masks, outdir=outdir,
filters=filters, s=f'{tract}_{patch}_{sp}', source_catalog=catalog)
print(f'\nSaved scarlet results as {filenames} \n')
[55]:
%%time
outdir='/home/g4merz/ddtutorials/'
generate_training_data_example(dirpath, tract='3828',patch='1,2',sp=18,plot_scene=True, plot_likelihood=False, write_results=True, outdir=outdir)
/tmp/ipykernel_1865293/2061951149.py:16: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_x'] = xs[inds]
/tmp/ipykernel_1865293/2061951149.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_y'] = ys[inds]
No flux in morphology model for source at (4.8566570126986335, 192.3978758655976)
No flux in morphology model for source at (156.3428668810957, 46.73144519767811)
Source catalog has 133 objects
Initializing starlet sources to be fit.
No flux in morphology model for source at (67.96124508293269, 238.5802424436115)
No flux in morphology model for source at (137.73664723412912, 235.1336232367812)
No flux in morphology model for source at (151.30783233206603, 15.49913299020045)
No flux in morphology model for source at (205.80671436111697, 69.20862945785666)
No flux in morphology model for source at (254.45607877980729, 254.17204999562637)
No flux in morphology model for source at (254.82916063899484, 193.02193773697945)
No flux in morphology model for source at (190.15911361222243, 96.36605434686498)
No flux in morphology model for source at (220.00362330847292, 34.43471952795517)
No flux in morphology model for source at (73.34399792161366, 203.80719093122207)
No flux in morphology model for source at (66.2767218112549, 223.11999092828228)
No flux in morphology model for source at (146.0457846056479, 196.70802377106156)
No flux in morphology model for source at (11.427058219639548, 91.917219322655)
No flux in morphology model for source at (8.624658592711967, 49.616198645699114)
No flux in morphology model for source at (164.72532345692525, 159.95621277789724)
No flux in morphology model for source at (180.5357546699679, 132.4014173361138)
No flux in morphology model for source at (70.05972541433039, 4.617009766165211)
No flux in morphology model for source at (94.78623533690006, 187.95738084927507)
No flux in morphology model for source at (146.5371041736753, 176.06116346082672)
No flux in morphology model for source at (63.93866812968008, 19.978271651849354)
No flux in morphology model for source at (112.84423633361166, 12.858196012799453)
No flux in morphology model for source at (177.26033195369746, 205.5045591178059)
No flux in morphology model for source at (68.77579145182699, 240.10354274096971)
No flux in morphology model for source at (212.08047877016907, 255.59589103198232)
No flux in morphology model for source at (63.79891027631675, 138.5052853766556)
No flux in morphology model for source at (220.6454541773901, 87.11451980020865)
No flux in morphology model for source at (208.82136830569198, 189.5105806624906)
No flux in morphology model for source at (19.174231350117225, 164.62132388394457)
No flux in morphology model for source at (241.59759671732263, 143.5272640550338)
No flux in morphology model for source at (55.27558236801087, 141.87193512030717)
No flux in morphology model for source at (236.20709096251267, 88.48014344568037)
No flux in morphology model for source at (151.55589008909374, 221.82094148116994)
No flux in morphology model for source at (23.1230025501809, 194.11628132846454)
No flux in morphology model for source at (63.183660477203375, 212.22054584499529)
No flux in morphology model for source at (68.69260822390152, 2.443093195579422)
No flux in morphology model for source at (78.57289334103007, 34.30851295291177)
No flux in morphology model for source at (50.72063670119496, 36.54722320501969)
No flux in morphology model for source at (185.8973042287971, 46.41161300408021)
No flux in morphology model for source at (66.60533197400764, 14.587325872662404)
No flux in morphology model for source at (141.6912315239697, 171.47337046452958)
No flux in morphology model for source at (55.81436281324659, 20.88667485224323)
No flux in morphology model for source at (41.269971756698396, 216.91149449003024)
No flux in morphology model for source at (232.08941497377418, 60.613665288941775)
No flux in morphology model for source at (71.3352083877644, 185.15200677251232)
No flux in morphology model for source at (42.627515175177905, 203.84096579586185)
No flux in morphology model for source at (5.533795563123022, 119.53272075219502)
No flux in morphology model for source at (236.3807075031027, 67.22606238573462)
No flux in morphology model for source at (18.151089916093042, 96.67057179932817)
No flux in morphology model for source at (260.200387452448, 123.42532026644221)
No flux in morphology model for source at (180.5914395292275, 100.39884296950186)
No flux in morphology model for source at (60.2990395510933, 43.604490402800366)
No flux in morphology model for source at (84.12656182105002, 128.63780444584154)
No flux in morphology model for source at (68.48056310821175, 156.4851438158912)
No flux in morphology model for source at (102.46337322601175, 51.781862922609434)
No flux in morphology model for source at (12.893696388772696, 18.246777937049046)
No flux in morphology model for source at (185.94957807500123, 184.02580161579863)
No flux in morphology model for source at (221.2778475779278, 101.6276959655188)
No flux in morphology model for source at (226.10298496714768, 162.56793060726886)
No flux in morphology model for source at (205.39162085979387, 167.1606127341238)
No flux in morphology model for source at (221.81352901961054, 128.06189241769243)
No flux in morphology model for source at (21.225375112724578, 169.6800178574158)
No flux in morphology model for source at (121.73589463679036, 7.858915810524195)
No flux in morphology model for source at (17.9398731041083, 80.58771611166776)
No flux in morphology model for source at (56.71680974521951, 82.95770253851879)
No flux in morphology model for source at (220.09468193997327, 243.54505119862733)
No flux in morphology model for source at (214.03426161226162, 101.68355426868402)
No flux in morphology model for source at (34.33474253463737, 111.85912210750575)
No flux in morphology model for source at (156.37820677392938, 67.00480758376943)
No flux in morphology model for source at (95.03509390387171, 174.84134570000606)
No flux in morphology model for source at (96.44074523891686, 150.30024482251974)
No flux in morphology model for source at (78.91834161502629, 68.0983395512958)
No flux in morphology model for source at (144.96414625583384, 232.8081601819049)
No flux in morphology model for source at (179.26247158983006, 45.4649691708637)
No flux in morphology model for source at (195.03851971786116, 15.67225634028182)
No flux in morphology model for source at (194.39868861830564, 254.05152151742368)
No flux in morphology model for source at (4.093311401587016, 178.55986860962366)
No flux in morphology model for source at (113.4675208189974, 59.826090897386166)
No flux in morphology model for source at (2.4107261346953237, 52.594063377597195)
No flux in morphology model for source at (97.5803685267847, 252.82720439827426)
No flux in morphology model for source at (85.6509333906115, 168.8695142619381)
No flux in morphology model for source at (14.468926824557457, 133.90256503371893)
No flux in morphology model for source at (11.668808177023493, 259.70202848958434)
No flux in morphology model for source at (21.047329659045317, 249.7890530706136)
No flux in morphology model for source at (98.43263226170347, 202.68922455618667)
No flux in morphology model for source at (19.41456904645929, 58.61111964200427)
No flux in morphology model for source at (169.57255392596653, 244.59612859358094)
No flux in morphology model for source at (200.84379311081102, 149.51582705325018)
No flux in morphology model for source at (55.36839460628926, 29.468064341694117)
No flux in morphology model for source at (37.340075206728216, 82.93586507036889)
No flux in morphology model for source at (119.94403896795575, 132.74828790127685)
No flux in morphology model for source at (131.7212808799286, 110.75076270739555)
No flux in morphology model for source at (150.75590924319386, 44.128696590982145)
No flux in morphology model for source at (112.87384387262136, 111.90533294918714)
No flux in morphology model for source at (32.865043376095855, 63.849859791422205)
No flux in morphology model for source at (187.5321567010078, 119.9741144617492)
No flux in morphology model for source at (140.58000581253054, 11.084511695942638)
No flux in morphology model for source at (99.3832594989617, 194.0607882649365)
No flux in morphology model for source at (191.3832350324392, 210.09756630667653)
No flux in morphology model for source at (111.60933628200382, 219.70719610419292)
No flux in morphology model for source at (103.98787347951202, 222.69037555093564)
No flux in morphology model for source at (228.0492283451449, 255.45093109525988)
No flux in morphology model for source at (164.47916394361255, 178.630141706215)
No flux in morphology model for source at (108.2017985616194, 203.95088967410084)
No flux in morphology model for source at (112.97636393563062, 206.593152055073)
No flux in morphology model for source at (185.53683345146965, 196.51927053578038)
No flux in morphology model for source at (5.998240542248823, 96.29761376045462)
No flux in morphology model for source at (228.71852762892104, 27.678966809184203)
No flux in morphology model for source at (197.04180964380066, 197.28413646178524)
Fitting Blend model.
Scarlet ran for 12 iterations to logL = -382118.035420464
18.239954710006714
Extracting deblended catalog.
Saved scarlet results as {'img_U': '/home/g4merz/ddtutorials/U_3828_1,2_18_scarlet_img.fits', 'img_G': '/home/g4merz/ddtutorials/G_3828_1,2_18_scarlet_img.fits', 'img_R': '/home/g4merz/ddtutorials/R_3828_1,2_18_scarlet_img.fits', 'img_I': '/home/g4merz/ddtutorials/I_3828_1,2_18_scarlet_img.fits', 'img_Z': '/home/g4merz/ddtutorials/Z_3828_1,2_18_scarlet_img.fits', 'img_Y': '/home/g4merz/ddtutorials/Y_3828_1,2_18_scarlet_img.fits', 'segmask': '/home/g4merz/ddtutorials/3828_1,2_18_scarlet_segmask.fits'}
CPU times: user 28.4 s, sys: 210 ms, total: 28.6 s
Wall time: 26.3 s
Let’s run a few more times to get a set of images
[61]:
generate_training_data_example(dirpath, tract='3828',patch='1,2',sp=17,plot_scene=False, plot_likelihood=False, write_results=True, outdir=outdir)
/tmp/ipykernel_1865293/2061951149.py:16: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_x'] = xs[inds]
/tmp/ipykernel_1865293/2061951149.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_y'] = ys[inds]
No flux in morphology model for source at (218.3070109456812, 99.2416584235325)
Source catalog has 109 objects
Initializing starlet sources to be fit.
No flux in morphology model for source at (94.67594995561558, 225.1134857168836)
No flux in morphology model for source at (49.09945510661055, 193.20271229321042)
No flux in morphology model for source at (248.86241752741444, 70.40285406602743)
No flux in morphology model for source at (115.70031321197985, 188.64343529850157)
No flux in morphology model for source at (64.48731036841855, 125.63588542193065)
No flux in morphology model for source at (1.044872974847749, 182.987356362617)
No flux in morphology model for source at (51.998881765652186, 94.56574516232467)
No flux in morphology model for source at (93.01757206149341, 228.9946473001437)
No flux in morphology model for source at (22.38093496411966, 4.981205480895369)
No flux in morphology model for source at (57.519048406034926, 64.59995863480253)
No flux in morphology model for source at (110.41891163182754, 9.894160240013662)
No flux in morphology model for source at (245.17932436185765, 73.62393713359779)
No flux in morphology model for source at (118.9378457467028, 38.63570170494677)
No flux in morphology model for source at (125.204434314237, 77.24351093796759)
No flux in morphology model for source at (144.79950671219376, 70.1616757047941)
No flux in morphology model for source at (149.04874357461722, 197.10991456960073)
No flux in morphology model for source at (69.6598814384406, 135.23853470267932)
No flux in morphology model for source at (82.43237451499135, 213.8087676921532)
No flux in morphology model for source at (92.1023599635173, 224.9445337148063)
No flux in morphology model for source at (195.36952737198862, 85.990026766136)
No flux in morphology model for source at (85.79788969983747, 109.4050533651025)
No flux in morphology model for source at (95.42843558821096, 254.86176343092848)
No flux in morphology model for source at (3.196528269542796, 193.9351648330794)
No flux in morphology model for source at (193.34134172064932, 63.231807887774266)
No flux in morphology model for source at (82.91385030171568, 103.61266434687786)
No flux in morphology model for source at (129.7361355327257, 204.67892450030376)
No flux in morphology model for source at (42.891583092052315, 0.9350918650434323)
No flux in morphology model for source at (225.53691183594128, 193.42253792033625)
No flux in morphology model for source at (240.31885593949664, 176.57782140432028)
No flux in morphology model for source at (215.58653449219037, 227.9453867655684)
No flux in morphology model for source at (23.24760857380079, 135.84789632016145)
No flux in morphology model for source at (184.6516653127028, 70.88340254288778)
No flux in morphology model for source at (123.06529905124353, 166.31642759741226)
No flux in morphology model for source at (40.94465217564357, 114.7720833852909)
No flux in morphology model for source at (57.37353968484058, 120.40487548178135)
No flux in morphology model for source at (195.77347339230164, 56.766373180891605)
No flux in morphology model for source at (247.90632231928612, 227.46263235130573)
No flux in morphology model for source at (117.46504121577527, 80.42216252225626)
No flux in morphology model for source at (217.4488044981863, 149.8425945548879)
No flux in morphology model for source at (108.54921817661852, 22.52071558206626)
No flux in morphology model for source at (61.351539960640366, 134.3483152236331)
No flux in morphology model for source at (20.898638602446226, 55.465633614405306)
No flux in morphology model for source at (77.56513774431642, 142.0089786587414)
No flux in morphology model for source at (98.9081481990761, 248.9225978809245)
No flux in morphology model for source at (259.4593536263301, 23.922642716539485)
No flux in morphology model for source at (246.12361905845955, 37.46414029388143)
No flux in morphology model for source at (62.2822136406794, 242.54540129467568)
No flux in morphology model for source at (120.45840728236271, 66.36709576342037)
No flux in morphology model for source at (203.9925413661167, 43.180571777364094)
No flux in morphology model for source at (110.80859316180158, 219.65772631294385)
No flux in morphology model for source at (131.6335082211981, 206.30164595427232)
No flux in morphology model for source at (105.63356523195034, 194.49493415668985)
No flux in morphology model for source at (138.88010625226616, 187.02439124741431)
No flux in morphology model for source at (157.01254645493373, 184.1903768240336)
No flux in morphology model for source at (225.03487497431433, 191.8686556048142)
No flux in morphology model for source at (25.112091746302212, 213.7709708514576)
No flux in morphology model for source at (177.5424343539653, 187.7998717712544)
No flux in morphology model for source at (115.4798872327683, 136.15592596842362)
No flux in morphology model for source at (181.95735968451208, 79.57757678974667)
No flux in morphology model for source at (30.585309329112533, 21.010134710435523)
No flux in morphology model for source at (198.5810722232909, 101.95346351350963)
No flux in morphology model for source at (150.72304418537988, 259.08578851474886)
No flux in morphology model for source at (87.86922034518557, 73.42326406457141)
No flux in morphology model for source at (9.726895673553372, 180.55370476860116)
No flux in morphology model for source at (41.074860244610136, 82.74747390260927)
No flux in morphology model for source at (218.25072956219537, 216.1458496790474)
No flux in morphology model for source at (242.8190298468071, 158.9102822435816)
No flux in morphology model for source at (170.31974377520055, 175.21888429772844)
No flux in morphology model for source at (187.12724882808197, 161.69165889840042)
No flux in morphology model for source at (49.79167749886983, 257.9338755171266)
No flux in morphology model for source at (85.87036265931238, 1.7891920978217968)
No flux in morphology model for source at (114.86930152170044, 191.76436373885917)
No flux in morphology model for source at (107.60001960874433, 174.14703540535265)
No flux in morphology model for source at (32.43988589313631, 129.1836785995256)
No flux in morphology model for source at (211.1326708045035, 250.70054245062056)
No flux in morphology model for source at (217.1274204841502, 217.80420824038083)
No flux in morphology model for source at (118.21424364316226, 226.90750264939743)
No flux in morphology model for source at (125.71712030343951, 29.35584300590017)
No flux in morphology model for source at (139.49481123116857, 50.745303875912214)
No flux in morphology model for source at (238.10917811306172, 254.54300302709635)
No flux in morphology model for source at (100.71594407763314, 5.422539027294988)
No flux in morphology model for source at (116.64881260462062, 259.06953576870546)
No flux in morphology model for source at (156.52780188397992, 180.1942513761096)
No flux in morphology model for source at (255.36496070817248, 199.14198189240233)
No flux in morphology model for source at (71.86362617602663, 56.97748463856078)
No flux in morphology model for source at (178.5414275849089, 98.77094051024869)
No flux in morphology model for source at (79.03907347463701, 178.35593930783216)
No flux in morphology model for source at (30.609960817988394, 108.50577107279969)
No flux in morphology model for source at (91.20844009158918, 166.11373307191025)
No flux in morphology model for source at (191.92678971837086, 143.03208909963178)
No flux in morphology model for source at (239.93773083866472, 140.02490986071825)
No flux in morphology model for source at (209.9168394051385, 199.16768591682558)
No flux in morphology model for source at (123.69568531000459, 99.37724477518168)
No flux in morphology model for source at (12.809698712387217, 117.5669498175539)
No flux in morphology model for source at (12.776269877826053, 50.02552433465826)
No flux in morphology model for source at (137.60336861024552, 111.01579156278058)
No flux in morphology model for source at (233.66038257033597, 98.48402350037759)
No flux in morphology model for source at (13.81870710739804, 219.94607263811486)
No flux in morphology model for source at (57.412678535926716, 205.61230890634943)
Fitting Blend model.
Scarlet ran for 5 iterations to logL = -382214.71846933855
10.130998373031616
Extracting deblended catalog.
Saved scarlet results as {'img_U': '/home/g4merz/ddtutorials/U_3828_1,2_17_scarlet_img.fits', 'img_G': '/home/g4merz/ddtutorials/G_3828_1,2_17_scarlet_img.fits', 'img_R': '/home/g4merz/ddtutorials/R_3828_1,2_17_scarlet_img.fits', 'img_I': '/home/g4merz/ddtutorials/I_3828_1,2_17_scarlet_img.fits', 'img_Z': '/home/g4merz/ddtutorials/Z_3828_1,2_17_scarlet_img.fits', 'img_Y': '/home/g4merz/ddtutorials/Y_3828_1,2_17_scarlet_img.fits', 'segmask': '/home/g4merz/ddtutorials/3828_1,2_17_scarlet_segmask.fits'}
[62]:
generate_training_data_example(dirpath, tract='3828',patch='1,2',sp=16,plot_scene=False, plot_likelihood=False, write_results=True, outdir=outdir)
/tmp/ipykernel_1865293/2061951149.py:16: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_x'] = xs[inds]
/tmp/ipykernel_1865293/2061951149.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_y'] = ys[inds]
No flux in morphology model for source at (34.215208205953786, 217.35673465162472)
Source catalog has 119 objects
Initializing starlet sources to be fit.
No flux in morphology model for source at (193.11070345324515, 65.12487340850748)
No flux in morphology model for source at (221.05143185157885, 0.7492544866054232)
No flux in morphology model for source at (101.28824767350216, 238.23671345504408)
No flux in morphology model for source at (233.49110628210474, 110.32720059355597)
No flux in morphology model for source at (86.97042768460051, 70.51089864074856)
No flux in morphology model for source at (200.2937977193551, 90.03842204550347)
No flux in morphology model for source at (207.1677354178546, 135.52044369708892)
No flux in morphology model for source at (105.05044080479456, 55.38957839943032)
No flux in morphology model for source at (61.06991255288176, 86.05271477497627)
No flux in morphology model for source at (230.54696021899326, 118.65910918205009)
No flux in morphology model for source at (245.34923312978117, 19.674303958017845)
No flux in morphology model for source at (136.86152141165167, 28.331969867869702)
No flux in morphology model for source at (258.2241811778358, 68.53390419356765)
No flux in morphology model for source at (15.209587929271947, 122.51834204199622)
No flux in morphology model for source at (141.24571132762867, 245.33064001364255)
No flux in morphology model for source at (224.61190981768686, 182.97589334408258)
No flux in morphology model for source at (217.0570263132895, 17.094995778354132)
No flux in morphology model for source at (79.28207548755927, 217.4451017197389)
No flux in morphology model for source at (168.61012748485064, 180.52463426771828)
No flux in morphology model for source at (122.31873278268631, 126.32892029609138)
No flux in morphology model for source at (228.9485275575098, 110.29752509198988)
No flux in morphology model for source at (158.37769685454896, 183.88665613685953)
No flux in morphology model for source at (240.86540034815062, 105.88606334837823)
No flux in morphology model for source at (100.2764535938004, 84.09068547229617)
No flux in morphology model for source at (11.241049042804661, 45.102481771091334)
No flux in morphology model for source at (219.25681563512262, 116.21464651244423)
No flux in morphology model for source at (188.10445115581024, 74.4704973725693)
No flux in morphology model for source at (18.08868952678222, 148.76552466887915)
No flux in morphology model for source at (145.63344473808138, 83.23349845080702)
No flux in morphology model for source at (202.30026060105047, 65.88409890456569)
No flux in morphology model for source at (170.44499450460444, 67.88329813067321)
No flux in morphology model for source at (112.05369656424045, 217.8737594375907)
No flux in morphology model for source at (68.42154270122955, 59.42496809994009)
No flux in morphology model for source at (72.68836789888064, 31.4095126112461)
No flux in morphology model for source at (180.39922324841245, 47.20739140225851)
No flux in morphology model for source at (234.08160332722946, 246.1005353242108)
No flux in morphology model for source at (232.5533214639736, 217.31968944806977)
No flux in morphology model for source at (227.60712134876485, 65.5295332387177)
No flux in morphology model for source at (77.45059744635364, 38.24091479711751)
No flux in morphology model for source at (166.31578734771574, 121.51631188240572)
No flux in morphology model for source at (105.40479113668243, 20.59695602242209)
No flux in morphology model for source at (25.774411579978732, 222.57121844544236)
No flux in morphology model for source at (79.1638528604517, 253.23617465388452)
No flux in morphology model for source at (214.28699187371058, 48.865183784535475)
No flux in morphology model for source at (182.1062083621291, 107.36169886490461)
No flux in morphology model for source at (204.80941868206992, 23.711788452748806)
No flux in morphology model for source at (237.69425731401043, 23.80115826589099)
No flux in morphology model for source at (258.5866013847062, 152.79053987706357)
No flux in morphology model for source at (26.696284242689217, 24.434718442696976)
No flux in morphology model for source at (197.95663791059633, 105.70946965153234)
No flux in morphology model for source at (52.571541812151736, 125.65796593956293)
No flux in morphology model for source at (185.42775718495523, 51.52389623853014)
No flux in morphology model for source at (108.71105377803633, 45.21093263591683)
No flux in morphology model for source at (109.76397555913718, 24.469522111090555)
No flux in morphology model for source at (182.4977310345348, 85.30251159054569)
No flux in morphology model for source at (71.04181339264505, 227.36248666106258)
No flux in morphology model for source at (245.7527303562656, 221.6822546596286)
No flux in morphology model for source at (165.49504763158166, 115.03959873185522)
No flux in morphology model for source at (243.96985189881525, 85.80117113257256)
No flux in morphology model for source at (259.9526043996975, 18.2385093841076)
No flux in morphology model for source at (117.50120679697375, 127.55994850557181)
No flux in morphology model for source at (140.00276968510843, 16.03449107535198)
No flux in morphology model for source at (233.42571030845465, 5.811444800247045)
No flux in morphology model for source at (33.200586152765936, 118.53024319680844)
No flux in morphology model for source at (201.86531177339566, 57.13616127398927)
No flux in morphology model for source at (190.28923221354398, 81.30551837623716)
No flux in morphology model for source at (39.794546698163686, 168.43028770408273)
No flux in morphology model for source at (83.70332281778428, 14.123794689514398)
No flux in morphology model for source at (176.5038493555885, 38.55082494452654)
No flux in morphology model for source at (104.12391241388832, 117.13001124268158)
No flux in morphology model for source at (17.47327711067919, 179.91613641343065)
No flux in morphology model for source at (198.43393635433404, 13.633086440564512)
No flux in morphology model for source at (41.3906454337457, 109.62363720595204)
No flux in morphology model for source at (120.00029403866756, 84.32980066360687)
No flux in morphology model for source at (5.525409898355065, 42.92985542633505)
No flux in morphology model for source at (205.69228378036405, 0.10368407310670591)
No flux in morphology model for source at (209.22467501250958, 91.24280517203442)
No flux in morphology model for source at (119.57540506516034, 31.89912873733556)
No flux in morphology model for source at (157.2470412659959, 90.14601803725054)
No flux in morphology model for source at (123.10395486899688, 51.40883069087431)
No flux in morphology model for source at (241.99413422961788, 130.90806259507917)
No flux in morphology model for source at (252.05521031253193, 49.12449944379114)
No flux in morphology model for source at (181.20984912037147, 209.47276199217413)
No flux in morphology model for source at (249.02347080306936, 17.964606706365885)
No flux in morphology model for source at (195.88660147876635, 76.83213167777103)
No flux in morphology model for source at (36.21964758838021, 37.30237087808564)
No flux in morphology model for source at (86.0752011360546, 78.75247709337964)
No flux in morphology model for source at (101.4105570988213, 112.07313844134296)
No flux in morphology model for source at (180.8966036818174, 139.34017224344643)
No flux in morphology model for source at (241.1296681780059, 249.42365251852607)
No flux in morphology model for source at (234.31902213602189, 205.34709212553753)
No flux in morphology model for source at (161.3138407583392, 239.07156547894738)
No flux in morphology model for source at (49.504463696735, 51.04674261576656)
No flux in morphology model for source at (122.22949909769068, 108.95664582648715)
No flux in morphology model for source at (137.5506415916807, 105.49146969070534)
No flux in morphology model for source at (124.53486069051542, 253.9702709906178)
No flux in morphology model for source at (86.01387900497957, 64.88926441901276)
No flux in morphology model for source at (168.7404432743142, 8.314433597766765)
No flux in morphology model for source at (234.7011717366404, 61.093923049287696)
No flux in morphology model for source at (83.63808771894855, 213.73433593421214)
No flux in morphology model for source at (169.44251232267925, 39.53399081356292)
Fitting Blend model.
Scarlet ran for 8 iterations to logL = -382418.8259674484
12.833014965057373
Extracting deblended catalog.
Saved scarlet results as {'img_U': '/home/g4merz/ddtutorials/U_3828_1,2_16_scarlet_img.fits', 'img_G': '/home/g4merz/ddtutorials/G_3828_1,2_16_scarlet_img.fits', 'img_R': '/home/g4merz/ddtutorials/R_3828_1,2_16_scarlet_img.fits', 'img_I': '/home/g4merz/ddtutorials/I_3828_1,2_16_scarlet_img.fits', 'img_Z': '/home/g4merz/ddtutorials/Z_3828_1,2_16_scarlet_img.fits', 'img_Y': '/home/g4merz/ddtutorials/Y_3828_1,2_16_scarlet_img.fits', 'segmask': '/home/g4merz/ddtutorials/3828_1,2_16_scarlet_segmask.fits'}
[ ]:
Annotation generation
Now we can reformat the scarlet output to the format that detectron2 (and thus deepdisc) expects. This is done by creating a list of dictionaries, with each dict containing annotations that have the ground truth information. Take a look at the detectron2 docs for more information on the format expected
[63]:
from deepdisc.data_format.file_io import DDLoader
from deepdisc.data_format.annotation_functions.annotate_dc2 import annotate_dc2
Each image cutout is saved into separate FITS files per band, along with mask files (which contain the object location/footprint information). Here we create a DDLoader class, which helps gather output files and format them
[19]:
%%time
outdir='./'
nb=8
generate_training_data_example(dirpath, tract='3828',patch='1,2',sp=4,plot_scene=False, plot_likelihood=False, write_results=True, outdir=outdir)
3828 1,2 4
(6, 4200, 4200)
[525, 525]
/tmp/ipykernel_3239168/2061951149.py:16: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_x'] = xs[inds]
/tmp/ipykernel_3239168/2061951149.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
dcut['new_y'] = ys[inds]
Source catalog has 182 objects
Initializing starlet sources to be fit.
No flux in morphology model for source at (209.72048181298032, 267.9881317471154)
No flux in morphology model for source at (188.5293816510548, 97.34488385190161)
No flux in morphology model for source at (358.268929609163, 509.186786307735)
No flux in morphology model for source at (175.53232860311982, 121.8747957374926)
No flux in morphology model for source at (24.594526692611907, 103.16194218067358)
No flux in morphology model for source at (399.01324522401774, 9.941506946473964)
No flux in morphology model for source at (396.6881047666475, 398.57640248103326)
No flux in morphology model for source at (497.8045912824191, 396.22481004178735)
No flux in morphology model for source at (391.2592264122586, 229.09565006552657)
No flux in morphology model for source at (18.388651363947247, 3.547778266623027)
No flux in morphology model for source at (405.2788180630714, 227.80441014273856)
No flux in morphology model for source at (81.76320703214515, 263.2321413857162)
No flux in morphology model for source at (260.48324288509866, 319.8606781092967)
No flux in morphology model for source at (420.6027919212238, 10.842692806425475)
No flux in morphology model for source at (92.00770149502387, 107.39299132491942)
No flux in morphology model for source at (189.30988451176927, 189.999427900897)
No flux in morphology model for source at (154.9908888261989, 520.2399896065317)
No flux in morphology model for source at (348.01203068562154, 274.83158087281936)
No flux in morphology model for source at (108.42519683941464, 5.385727065768151)
No flux in morphology model for source at (71.05555147588075, 116.12919722768402)
No flux in morphology model for source at (74.30619394614496, 225.51370327069708)
No flux in morphology model for source at (413.98821150496224, 316.4635388892402)
No flux in morphology model for source at (515.4139772733952, 150.97795200013388)
No flux in morphology model for source at (177.78283985007056, 99.779629139407)
No flux in morphology model for source at (192.45145828486966, 521.8478303167622)
No flux in morphology model for source at (195.60894458937037, 324.8843048184117)
No flux in morphology model for source at (492.6532879294091, 282.6984109575933)
No flux in morphology model for source at (275.7587326250732, 278.5729362114607)
No flux in morphology model for source at (98.07441322947034, 153.2902087441471)
No flux in morphology model for source at (87.46120664095542, 161.2061620014556)
No flux in morphology model for source at (319.85075005841554, 421.5296920813644)
No flux in morphology model for source at (118.94830067059775, 223.38824790703075)
No flux in morphology model for source at (275.7646490788784, 384.8880796036083)
No flux in morphology model for source at (284.0186830881048, 431.9091063710466)
No flux in morphology model for source at (408.6648718572278, 124.14633201104243)
No flux in morphology model for source at (74.87217174062971, 98.5072945659831)
No flux in morphology model for source at (488.5419877541599, 431.867821967252)
No flux in morphology model for source at (458.44447630174, 241.16250924974793)
No flux in morphology model for source at (382.75481100111483, 506.16823202759406)
No flux in morphology model for source at (419.0100747483302, 362.24846226294176)
No flux in morphology model for source at (410.11716389339654, 318.0619133414157)
No flux in morphology model for source at (61.91220333066394, 99.45450848963173)
No flux in morphology model for source at (337.3767872244034, 370.3828250321576)
No flux in morphology model for source at (206.72226895307813, 349.8935454096427)
No flux in morphology model for source at (287.187915713389, 118.59274910522163)
No flux in morphology model for source at (500.36191445062286, 405.8729917768678)
No flux in morphology model for source at (205.22846425335047, 60.785669656296704)
No flux in morphology model for source at (243.3867825291427, 323.55429500824357)
No flux in morphology model for source at (81.42938091565793, 318.64035392234655)
No flux in morphology model for source at (209.67314350512333, 223.71827688394478)
No flux in morphology model for source at (109.63071989361106, 516.8907546445016)
No flux in morphology model for source at (450.2985471041602, 519.7276029231216)
No flux in morphology model for source at (416.9396022118599, 242.56926945539817)
No flux in morphology model for source at (43.28821028920811, 192.09579432687406)
No flux in morphology model for source at (307.33383964383484, 410.89706192983704)
No flux in morphology model for source at (364.7248753196436, 495.39870044297186)
No flux in morphology model for source at (519.0186493547371, 244.64309892076108)
No flux in morphology model for source at (340.759685762092, 354.69742031455826)
No flux in morphology model for source at (513.3956987702477, 319.8706579500449)
No flux in morphology model for source at (198.45766854819885, 432.1275045914572)
No flux in morphology model for source at (191.70266722788074, 175.65559038208266)
No flux in morphology model for source at (217.8145155238626, 397.5779604891495)
No flux in morphology model for source at (33.552365877995726, 177.4295790337883)
No flux in morphology model for source at (236.6737281727692, 192.29371376751442)
No flux in morphology model for source at (250.1089713045567, 106.8381049956206)
No flux in morphology model for source at (316.4953735719164, 448.4575949162563)
No flux in morphology model for source at (29.65972688833608, 210.2018058661315)
No flux in morphology model for source at (255.75599145047636, 384.64578576933945)
No flux in morphology model for source at (124.25113561390481, 453.79396942080166)
No flux in morphology model for source at (438.539898698903, 188.05134416426063)
No flux in morphology model for source at (320.4079198786512, 485.1392340596003)
No flux in morphology model for source at (408.7418711698756, 317.5461811548312)
No flux in morphology model for source at (382.7894646131499, 115.79891780264825)
No flux in morphology model for source at (71.10387202447782, 301.0627813126521)
No flux in morphology model for source at (306.31342488945484, 336.58323198864946)
No flux in morphology model for source at (430.999081322072, 435.96953981946535)
No flux in morphology model for source at (172.39813094366764, 46.35939123593471)
No flux in morphology model for source at (69.02448914653723, 489.60029548022885)
No flux in morphology model for source at (125.63911064672811, 90.92928428202958)
No flux in morphology model for source at (394.61894291003864, 433.468880926107)
No flux in morphology model for source at (18.40758934209225, 495.70325458394564)
No flux in morphology model for source at (335.09997934525927, 59.261559128212866)
No flux in morphology model for source at (71.7195076929147, 449.06520457765055)
No flux in morphology model for source at (36.64600022921786, 185.4936650888685)
No flux in morphology model for source at (266.302734816516, 377.91800388779393)
No flux in morphology model for source at (310.0561855906708, 9.110904537537863)
No flux in morphology model for source at (462.4467928902286, 450.6559869722987)
No flux in morphology model for source at (499.51756733564434, 489.6585248547199)
No flux in morphology model for source at (499.9900816969566, 353.2303908757467)
No flux in morphology model for source at (38.798845459322365, 195.84683424068953)
No flux in morphology model for source at (82.0636108971994, 85.36823998554064)
No flux in morphology model for source at (11.437873752141059, 200.98908014520384)
No flux in morphology model for source at (391.3763400391672, 441.47413244071595)
No flux in morphology model for source at (121.09217143103524, 486.93066483612165)
No flux in morphology model for source at (338.0884296755348, 373.09613041106604)
No flux in morphology model for source at (382.8458595891252, 346.84567032771247)
No flux in morphology model for source at (207.04199077075737, 177.1725771363299)
No flux in morphology model for source at (120.21339730121235, 84.55965133351037)
No flux in morphology model for source at (382.85551512953407, 85.44420050894132)
No flux in morphology model for source at (194.4900949915209, 517.263936667905)
No flux in morphology model for source at (426.7794970629802, 446.80055933538733)
No flux in morphology model for source at (221.64338200006296, 152.2002183444356)
No flux in morphology model for source at (310.52630445337036, 410.554161420775)
No flux in morphology model for source at (132.12729820134518, 5.425129389757785)
No flux in morphology model for source at (47.28743770341407, 449.0534038609776)
No flux in morphology model for source at (473.10573628983275, 523.4771203143191)
No flux in morphology model for source at (475.0083280256831, 431.8906883921509)
No flux in morphology model for source at (443.5326652412514, 508.34738872015714)
No flux in morphology model for source at (192.03128285114053, 49.10817996730475)
Fitting Blend model.
Scarlet ran for 15 iterations to logL = -2321654.6215960835
69.64574384689331
Extracting deblended catalog.
Saved scarlet results as {'img_U': './U_3828_1,2_4_scarlet_img.fits', 'img_G': './G_3828_1,2_4_scarlet_img.fits', 'img_R': './R_3828_1,2_4_scarlet_img.fits', 'img_I': './I_3828_1,2_4_scarlet_img.fits', 'img_Z': './Z_3828_1,2_4_scarlet_img.fits', 'img_Y': './Y_3828_1,2_4_scarlet_img.fits', 'segmask': './3828_1,2_4_scarlet_segmask.fits'}
CPU times: user 1min 27s, sys: 2.68 s, total: 1min 30s
Wall time: 1min 30s
We can split the set of produced cutout images/mask files into different datasets with the random_sample functionality
[ ]:
This will make a copy of the data, so we can delete the old images
[68]:
for file in glob.glob(os.path.join(outdir,'*.fits')):
os.remove(file)
Generate a new filedict for the new train directory
[71]:
loader = DDLoader().generate_filedict('/home/g4merz/ddtutorials/train/', ['U', 'G', 'R', 'I', 'Z','Y'], '*_scarlet_img.fits', '*_scarlet_segmask.fits')
filedict = loader.filedict
img_files = np.transpose([filedict[filt]["img"] for filt in filedict["filters"]])
[74]:
from deepdisc.data_format import conversions
from deepdisc.data_format import conversions
from deepdisc.data_format.conversions import fitsim_to_numpy, ddict_to_hdf5, fitsim_to_hdf5
from deepdisc.data_format.annotation_functions.annotate_dc2 import annotate_dc2
Format the files for deepdisc
[77]:
dataset_dicts=[]
dataset_dicts = loader.generate_dataset_dict(annotate_dc2).get_dataset()
Now dataset_dicts contains the formatted ground truth for deepdisc.
[80]:
from detectron2.utils.file_io import PathManager
dfile ='/home/g4merz/ddtutorials/train/train_dicts.json'
with PathManager.open(dfile, "w") as f:
json.dump(dataset_dicts, f)
It may also be useful to format the FITS images by scarlet into numpy .npy files. For convenience, this can be done using some conversion utilities
[81]:
from deepdisc.data_format.conversions import fitsim_to_numpy
[84]:
fitsim_to_numpy(img_files,'/home/g4merz/ddtutorials/train/')
Clean up the FITS files
[85]:
for file in glob.glob('/home/g4merz/ddtutorials/train/*.fits'):
os.remove(file)
Now you have everything you need to run DeepDISC!
[ ]: