Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
276.98708ms
FAIL (2025-07-14 16:29:35 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 277.015919ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
270.393126ms
FAIL (2025-07-14 16:28:36 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 270.424635ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
313.353061ms
FAIL (2025-07-14 16:28:17 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 313.393167ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
452.970089ms
FAIL (2025-07-14 16:27:43 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 452.998352ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
233.166002ms
FAIL (2025-07-14 16:27:24 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 233.212261ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
275.11049ms
FAIL (2025-07-14 16:26:29 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 275.142623ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
267.318232ms
FAIL (2025-07-14 16:25:40 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 267.342967ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
238.58834ms
FAIL (2025-07-14 16:24:57 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 238.663045ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
250.873812ms
FAIL (2025-07-14 16:24:33 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 250.951101ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
221.905003ms
FAIL (2025-07-14 16:24:09 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 221.930122ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
255.949006ms
FAIL (2025-07-14 16:23:30 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 255.98159ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
224.011319ms
FAIL (2025-07-14 16:22:23 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 224.06821ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
272.102693ms
FAIL (2025-07-14 16:21:47 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 272.123644ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
292.847498ms
FAIL (2025-07-14 16:21:23 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 292.878464ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
250.097228ms
FAIL (2025-07-14 16:20:31 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 250.119897ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
228.791948ms
FAIL (2025-07-14 16:19:26 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 228.826126ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
300.428626ms
FAIL (2025-07-14 16:18:47 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 300.455661ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
379.260447ms
FAIL (2025-07-14 16:18:00 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 379.30714ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
270.471175ms
FAIL (2025-07-14 16:17:29 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 270.501509ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
238.011222ms
FAIL (2025-07-14 16:16:44 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 238.03574ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
271.668753ms
FAIL (2025-07-14 16:16:27 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 271.718991ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
544.104663ms
FAIL (2025-07-14 16:15:24 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 544.140916ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
285.143092ms
FAIL (2025-07-14 16:14:35 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 285.18005ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
303.343118ms
FAIL (2025-07-14 16:13:31 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 303.373888ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
213.295821ms
FAIL (2025-07-14 16:12:28 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 213.324987ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
408.122558ms
FAIL (2025-07-14 16:12:08 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 408.157229ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
266.126385ms
FAIL (2025-07-14 16:10:59 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 266.1575ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
499.751849ms
FAIL (2025-07-14 16:10:21 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 499.80121ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
281.225727ms
FAIL (2025-07-14 16:10:05 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 281.248891ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
268.529312ms
FAIL (2025-07-14 16:09:39 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 268.555785ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
231.638711ms
FAIL (2025-07-14 16:08:44 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 231.684387ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
235.643983ms
FAIL (2025-07-14 16:08:10 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 235.670091ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
265.410449ms
FAIL (2025-07-14 16:07:12 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 265.466028ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
480.809257ms
FAIL (2025-07-14 16:06:33 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 480.843965ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
246.689779ms
FAIL (2025-07-14 16:05:47 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 246.719539ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
265.760295ms
FAIL (2025-07-14 16:05:17 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 265.793682ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
285.974384ms
FAIL (2025-07-14 16:04:57 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 286.034341ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
235.78695ms
FAIL (2025-07-14 16:04:16 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 235.9356ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
300.894602ms
FAIL (2025-07-14 16:03:53 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 300.982977ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
362.636209ms
FAIL (2025-07-14 16:03:23 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 362.661524ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000714323s
FAIL (2025-07-14 16:03:11 UTC)
orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Starting upload Jul 14 15:55:08.565241979 (510h43m35.491536651s) [upload=408757]: Done starting upload Jul 14 15:55:08.565400991 (510h43m35.491695678s) [upload=408757]: Uploading segments... Jul 14 15:55:08.565544163 (510h43m35.491838864s) [upload=408757]: Got next segment. Inline: true Jul 14 15:55:08.565552872 (510h43m35.491847572s) [upload=408757]: Waiting for error group managing segments... Jul 14 15:55:08.565634353 (510h43m35.491929047s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:55:08.565641501 (510h43m35.491936199s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:55:08.565645907 (510h43m35.491940602s) [upload=408757]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:55:08.811788722 (510h43m35.738083419s) [upload=408757]: Done uploading segments... Jul 14 15:55:08.811812270 (510h43m35.738106947s) [upload=408757]: Upload finished. err: <nil> Jul 14 15:56:05.100227897 (510h44m32.026522596s) [upload=408769]: Starting upload Jul 14 15:56:05.100312658 (510h44m32.026607363s) [upload=408769]: Done starting upload Jul 14 15:56:05.100408970 (510h44m32.026703651s) [upload=408769]: Uploading segments... Jul 14 15:56:05.100472989 (510h44m32.026767692s) [upload=408769]: Got next segment. Inline: true Jul 14 15:56:05.100481607 (510h44m32.026776311s) [upload=408769]: Waiting for error group managing segments... Jul 14 15:56:05.100545751 (510h44m32.02684045s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:05.100552351 (510h44m32.026847052s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:05.100556814 (510h44m32.026851531s) [upload=408769]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:05.182190284 (510h44m32.108484963s) [upload=408769]: Done uploading segments... Jul 14 15:56:05.182214367 (510h44m32.108509041s) [upload=408769]: Upload finished. err: <nil> Jul 14 15:56:23.561600009 (510h44m50.487894682s) [upload=408772]: Starting upload Jul 14 15:56:23.561697683 (510h44m50.487992369s) [upload=408772]: Done starting upload Jul 14 15:56:23.561763634 (510h44m50.488058306s) [upload=408772]: Uploading segments... Jul 14 15:56:23.561805939 (510h44m50.488100634s) [upload=408772]: Got next segment. Inline: true Jul 14 15:56:23.561810017 (510h44m50.488104688s) [upload=408772]: Waiting for error group managing segments... Jul 14 15:56:23.561847407 (510h44m50.488142079s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:56:23.561851632 (510h44m50.488146304s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:56:23.561854076 (510h44m50.488148748s) [upload=408772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:56:23.793680582 (510h44m50.719975274s) [upload=408772]: Done uploading segments... Jul 14 15:56:23.793703117 (510h44m50.719997812s) [upload=408772]: Upload finished. err: <nil> Jul 14 15:57:34.519878119 (510h46m1.446172811s) [upload=408788]: Starting upload Jul 14 15:57:34.519970480 (510h46m1.446265177s) [upload=408788]: Done starting upload Jul 14 15:57:34.520092411 (510h46m1.446387099s) [upload=408788]: Uploading segments... Jul 14 15:57:34.520166251 (510h46m1.446460944s) [upload=408788]: Got next segment. Inline: true Jul 14 15:57:34.520172205 (510h46m1.446466896s) [upload=408788]: Waiting for error group managing segments... Jul 14 15:57:34.520234530 (510h46m1.44652922s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:57:34.520242554 (510h46m1.446537246s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:57:34.520246414 (510h46m1.446541089s) [upload=408788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:57:34.591357941 (510h46m1.51765263s) [upload=408788]: Done uploading segments... Jul 14 15:57:34.591382458 (510h46m1.51767713s) [upload=408788]: Upload finished. err: <nil> Jul 14 15:58:21.181278288 (510h46m48.107572974s) [upload=408799]: Starting upload Jul 14 15:58:21.181414723 (510h46m48.10770941s) [upload=408799]: Done starting upload Jul 14 15:58:21.181504668 (510h46m48.107799344s) [upload=408799]: Uploading segments... Jul 14 15:58:21.181594057 (510h46m48.107888743s) [upload=408799]: Got next segment. Inline: true Jul 14 15:58:21.181599968 (510h46m48.107894655s) [upload=408799]: Waiting for error group managing segments... Jul 14 15:58:21.181655490 (510h46m48.107950176s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:58:21.181663575 (510h46m48.107958267s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:58:21.181666950 (510h46m48.107961639s) [upload=408799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:58:21.254313332 (510h46m48.180608024s) [upload=408799]: Done uploading segments... Jul 14 15:58:21.254337414 (510h46m48.180632092s) [upload=408799]: Upload finished. err: <nil> Jul 14 15:59:29.757582463 (510h47m56.683877164s) [upload=408811]: Starting upload Jul 14 15:59:29.757667050 (510h47m56.683961743s) [upload=408811]: Done starting upload Jul 14 15:59:29.757746038 (510h47m56.684040714s) [upload=408811]: Uploading segments... Jul 14 15:59:29.757796482 (510h47m56.68409118s) [upload=408811]: Got next segment. Inline: true Jul 14 15:59:29.757801821 (510h47m56.684096513s) [upload=408811]: Waiting for error group managing segments... Jul 14 15:59:29.757848310 (510h47m56.684143003s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 15:59:29.757855707 (510h47m56.684150406s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 15:59:29.757859794 (510h47m56.684154486s) [upload=408811]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 15:59:29.825119607 (510h47m56.751414299s) [upload=408811]: Done uploading segments... Jul 14 15:59:29.825142445 (510h47m56.751437135s) [upload=408811]: Upload finished. err: <nil> Jul 14 16:00:11.648463355 (510h48m38.574758053s) [upload=408818]: Starting upload Jul 14 16:00:11.648621094 (510h48m38.57491579s) [upload=408818]: Done starting upload Jul 14 16:00:11.648755961 (510h48m38.575050645s) [upload=408818]: Uploading segments... Jul 14 16:00:11.648856115 (510h48m38.575150812s) [upload=408818]: Got next segment. Inline: true Jul 14 16:00:11.648863578 (510h48m38.575158271s) [upload=408818]: Waiting for error group managing segments... Jul 14 16:00:11.648968244 (510h48m38.575262941s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:11.648976013 (510h48m38.575270714s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:11.648980488 (510h48m38.575275181s) [upload=408818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:11.720172712 (510h48m38.646467402s) [upload=408818]: Done uploading segments... Jul 14 16:00:11.720203386 (510h48m38.64649808s) [upload=408818]: Upload finished. err: <nil> Jul 14 16:00:39.684063309 (510h49m6.610358001s) [upload=408823]: Starting upload Jul 14 16:00:39.684197216 (510h49m6.610491914s) [upload=408823]: Done starting upload Jul 14 16:00:39.684330531 (510h49m6.610625206s) [upload=408823]: Uploading segments... Jul 14 16:00:39.684417335 (510h49m6.610712035s) [upload=408823]: Got next segment. Inline: true Jul 14 16:00:39.684424166 (510h49m6.610718864s) [upload=408823]: Waiting for error group managing segments... Jul 14 16:00:39.684496816 (510h49m6.610791513s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:00:39.684503739 (510h49m6.610798429s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:00:39.684507835 (510h49m6.610802532s) [upload=408823]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:00:39.773228188 (510h49m6.69952288s) [upload=408823]: Done uploading segments... Jul 14 16:00:39.773253855 (510h49m6.699548536s) [upload=408823]: Upload finished. err: <nil> Jul 14 16:01:22.423898202 (510h49m49.35019288s) [upload=408833]: Starting upload Jul 14 16:01:22.423960020 (510h49m49.3502547s) [upload=408833]: Done starting upload Jul 14 16:01:22.424032216 (510h49m49.350326893s) [upload=408833]: Uploading segments... Jul 14 16:01:22.424072024 (510h49m49.350366719s) [upload=408833]: Got next segment. Inline: true Jul 14 16:01:22.424077241 (510h49m49.350371918s) [upload=408833]: Waiting for error group managing segments... Jul 14 16:01:22.424119904 (510h49m49.35041458s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:01:22.424125156 (510h49m49.350419842s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:01:22.424128672 (510h49m49.350423359s) [upload=408833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:01:22.497137586 (510h49m49.423432279s) [upload=408833]: Done uploading segments... Jul 14 16:01:22.497176352 (510h49m49.423471058s) [upload=408833]: Upload finished. err: <nil> Jul 14 16:02:26.270217585 (510h50m53.196512262s) [upload=408846]: Starting upload Jul 14 16:02:26.270296932 (510h50m53.19659161s) [upload=408846]: Done starting upload Jul 14 16:02:26.270409290 (510h50m53.196703967s) [upload=408846]: Uploading segments... Jul 14 16:02:26.270448175 (510h50m53.196742869s) [upload=408846]: Got next segment. Inline: true Jul 14 16:02:26.270466493 (510h50m53.196761179s) [upload=408846]: Waiting for error group managing segments... Jul 14 16:02:26.270512654 (510h50m53.196807329s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Jul 14 16:02:26.270517616 (510h50m53.196812297s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Jul 14 16:02:26.270520833 (510h50m53.196815519s) [upload=408846]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Jul 14 16:02:26.384583512 (510h50m53.310878197s) [upload=408846]: Done uploading segments... Jul 14 16:02:26.384604960 (510h50m53.310899636s) [upload=408846]: Upload finished. err: <nil>
Duration: 30.000752467s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.122893675s
FAIL (2025-07-14 14:55:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.122992745s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
2.061865522s
FAIL (2025-07-14 14:53:55 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.06191736s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
2.043717s
FAIL (2025-07-14 14:53:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.043749367s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
4 out of 60 is failed (threshold=3) [XXXX........................................................] duration:
2.028464408s
FAIL (2025-07-14 14:53:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.028491945s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration:
2.048904548s
FAIL (2025-07-14 14:52:35 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.048933924s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration:
2.036854476s
FAIL (2025-07-14 14:51:53 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.036887174s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [.XXXXXX.....................................................] duration:
2.076044559s
FAIL (2025-07-14 14:51:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.076073912s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [..XXXXXX....................................................] duration:
2.081705327s
FAIL (2025-07-14 14:50:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.08173646s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [...XXXXXX...................................................] duration:
2.029496171s
FAIL (2025-07-14 14:49:10 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 21)
======LOGS======
pload=407322]: Uploading piece...
Jul 14 14:06:39.063588515 (508h55m5.989883212s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064077307 (508h55m5.990372104s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064374985 (508h55m5.990669683s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.064948360 (508h55m5.991243052s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065340873 (508h55m5.991635559s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.065684825 (508h55m5.991979513s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066052877 (508h55m5.992347564s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.066490096 (508h55m5.992784788s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Uploading piece...
Jul 14 14:06:39.503124227 (508h55m6.429418904s) [upload=407322]: Next returned nil segment
Jul 14 14:06:39.503143412 (508h55m6.429438101s) [upload=407322]: Waiting for error group managing segments...
Jul 14 14:06:39.657872002 (508h55m6.58416669s) [noise=true , address=81.171.14.70:4030 , piece_id=B2XER7VU2HNPSAR4CLL6QPZX2OQLMSWAEM3TMI3MOZCUJRXX7OVA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659098308 (508h55m6.585392987s) [noise=true , address=81.171.14.70:4040 , piece_id=MJCSM2N3VH3SMXTLVDAJTPFKRZYU6GU7VSF2QCEKLSJB2LIPY42A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.659956829 (508h55m6.586251508s) [noise=true , address=81.171.14.70:4034 , piece_id=TWFXTOIP4F23LANEYDKY63C5HHPRNNLZH5VTIU3PXQDPTXSQXDXQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661000220 (508h55m6.587294898s) [noise=true , address=81.171.14.70:4027 , piece_id=QAVBLELT6GWJM727SUAKGTPCGWS2J2GCGHFMME3UAVUZSPBYKZSQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.661974242 (508h55m6.588268924s) [noise=true , address=81.171.14.70:4036 , piece_id=AFQZIUC553CEHO4MTSP3SAWBOWCEFYEZMK63IF5SYS6TWUEY3KFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662536846 (508h55m6.588831534s) [noise=true , address=81.171.14.70:4043 , piece_id=IQPWEDULFGVN6YVODH6NKPLID3FNFDNP3TCT4QU7WU6FL66VVEQQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.662572795 (508h55m6.588867469s) [noise=true , address=81.171.14.70:4042 , piece_id=PFGHIMZBG4NK5RUJKQDMGAXKKUSG22Q6ESMSKCUGJSCMYAXKS5RA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.663334921 (508h55m6.589629598s) [noise=true , address=81.171.14.70:4032 , piece_id=RTLSZZO5YMRDXCXU4AATQUGGUZFPN7TUF6LLUONXCZH4GAE2ZP2A, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.667243363 (508h55m6.593538044s) [noise=true , address=81.171.14.70:4026 , piece_id=5MFL2BVO2FCT2DPE7KXSNRH7LU4N2YVE4FMI7WJPOYLB2MOE2IEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.668917672 (508h55m6.595212349s) [noise=true , address=81.171.14.70:4044 , piece_id=AX3I5BCPS5B6P37REY4ERPUXFGOO35AZWGTADYGFAKW4A3UFUKBA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669338702 (508h55m6.595633379s) [noise=true , address=185.167.204.206:4060 , piece_id=7OECDYGD4NNUA2DQZ7AUNGMK6RHI3OGGWMLB4BKCX3DLEMGHKYWA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.669684292 (508h55m6.595978984s) [noise=true , address=185.167.204.205:4018 , piece_id=LSHINT2ROLGM4BWTZ5YYRVNSTLB63QIT57HG2VMRK35G65B2QHIA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.670201354 (508h55m6.59649603s) [noise=true , address=185.167.204.205:4025 , piece_id=FEVVOL3RFYRWEVFWQKMUZ5QZOLCBH5VZZE5OOBW675A3JSZEAH7Q, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.673248377 (508h55m6.599543053s) [noise=true , address=81.171.14.70:4037 , piece_id=LNY7WGKVHK4UFAU5HSGPOVBFOAAJNQC7OH5YAZIOVKNUGZOS4QUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.675891192 (508h55m6.602185872s) [noise=true , address=185.167.204.205:4010 , piece_id=KHTKWVFCXCXKMZBL2MC6IKAA756D2QIQNYMDPZOUJIN4ABZ32ZDA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.676396580 (508h55m6.60269126s) [noise=true , address=81.171.14.70:4045 , piece_id=FW55YOHTLIGEID7AVGMMV6GW3SFHAHZMLOEWGWGQ7VY6LAPS3OLA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.678339205 (508h55m6.604633883s) [noise=true , address=185.167.204.206:4056 , piece_id=7UX5L5OCHLJKMYXFVD63KNXVHM6CFHDRATCWER57URZJNUDA4TOQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.679005794 (508h55m6.60530047s) [noise=true , address=185.167.204.205:4035 , piece_id=XV25KZCE7CKBFUTXWL3OLIML25YZBMQJOOV3QH6S432EFXMXINMA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.698372284 (508h55m6.624666961s) [noise=true , address=185.167.204.206:4038 , piece_id=FFUVTIBZCYPQ7UJV7JFX3CGGCJNFNMFEZ2TSMOK2VXHD7KGH5FTA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703166855 (508h55m6.629461539s) [noise=true , address=185.167.204.205:4024 , piece_id=R52CYEYEWUNQKI5DJIPEGDKQNDEGYK5DVK5J7YFQX3OJPZN4J7OQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:06:39.703250728 (508h55m6.629545408s) [noise=true , address=185.167.204.206:4062 , piece_id=7EQDVAAJO34F5ULV6YH2AKQ3ZKRWFP34C74KBT5RIPLL5EFAKGIQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: <nil>
Jul 14 14:07:38.558211992 (508h56m5.484506682s) [noise=true , address=89.149.201.147:4011 , piece_id=ZO2ABM6Z3VNZCBJVPN6HWP2GSEGGNAUSBW7DO4QNYWYI56UYFXFA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: rpc: dial tcp 89.149.201.147:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558344578 (508h56m5.484639268s) [noise=true , address=89.149.201.147:4023 , piece_id=CASSNMJXA6JJ2ERPOMEO2ZX3XAONK7VHFWWWEF3QSTNVWO5O3ISQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv, address:89.149.201.147:4023): protocol: context deadline exceeded; context deadline exceeded
Jul 14 14:07:38.558410726 (508h56m5.484705399s) [noise=true , address=89.149.201.147:4017 , piece_id=ZLBWRR7PE3NXDFY7LLQ7IVV2ZX2QS6EUFUWXK3HTNZJTI3TJPSAA, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: rpc: dial tcp 89.149.201.147:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558466084 (508h56m5.484760758s) [noise=true , address=89.149.201.147:4021 , piece_id=J6WYDPSFX33LQV6QXN4AYCCNT5B4426ZIZIPIN6WVH4ICMQB4SAQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: rpc: dial tcp 89.149.201.147:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558535040 (508h56m5.484829712s) [noise=true , address=89.149.201.147:4001 , piece_id=KB4CNAVBMAD7K7LP2J43PLZZN453TMK2XJQC6CBT6IHEENLWB5XQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: rpc: dial tcp 89.149.201.147:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558602218 (508h56m5.484896892s) [noise=true , address=89.149.201.147:4015 , piece_id=GFT74DS2LLBV637PNNJX4C2NFCUQ3M72CMTZYJBI6LPAWRWWUJEQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: rpc: dial tcp 89.149.201.147:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558658030 (508h56m5.484952721s) [noise=true , address=89.149.201.147:4012 , piece_id=FFPSMLIH34K6RTL54Y5KSMAJPAIBB2VSVOAYGXWNFCBNPJG77CUQ, seg_pos={0 0}, upload=407322]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: write tcp 159.223.143.39:33320->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 14 14:07:38.558680059 (508h56m5.484974757s) [seg_pos={0 0}, upload=407322]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil>
Jul 14 14:07:38.657743564 (508h56m5.584038249s) [upload=407322]: Done uploading segments...
Jul 14 14:07:38.657779742 (508h56m5.584074443s) [upload=407322]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21)
Duration: 2.029527611s
core
uplink
eu-select-1
eu1-placement-15
|